OnlineHD: Robust, Efficient, and Single-Pass Online Learning Using Hyperdimensional System

2024-02-28

Keywords: #HDC

---

0. Abstract

• HDC supports single-pass learning
• Limitations of single-pass learning: Weak classification accuracy due to model saturation caused by naively accumulating high-dimensional data.
• OnlineHD: For each data point, updates the model depending on how similar it is to the existing model, instead of naive data accumulation.

1. Introduction

• Why use HDC?
  • HDC: Based on the fact that the brain works with neural activities in high-dimensional space. It maps data points into high-dimensional space and then perform a nearly-linear training to learn a model.
    1. HDC models are computationally efficient and highly parallel at heart to train and amenable to hardware level optimization
    2. HDC models provide strong robustness to noise
• Problem: HDC single-pass training has low classification accuracy → HDC retraining → removes advantages that single-pass model provides, requiring large off-chip memory
• Why does HDC single-pass training lead to low classification accuracy? → Naive data accumulation to generate each class hypervector. This causes forgetting(?) and saturation in each class hypervector, where the pattern of common data dominates each class.
• Proposal: OnlineHD supports single-pass training while ensuring accuracy comparable to the retrained model.
• Contribution
  • During single-pass training, OnlineHD identifies common patterns in each class hypervector and eliminates model saturation. For data points with high similarity, OnlineHD gives very small weights to them during the model update, while dissimilar data points get larger weights depending on how far they are to the current model. OnlineHD provides comparable accuracy to the retrained model while getting all benefits that a single-pass model provides.
  • We study the impact of different data representation on OnlineHD robustness to possible noise in the hardware.

2. Hyper-Dimensional Classification

---