IEEE Circuits and Systems Magazine - Q2 2020 - 32

- ethods achieve high accuracy using complex models.
m
Training these models typically takes longer time and
requires more energy consumption. The models in HD
classification are simpler and can be trained in less
time with high energy efficiency. However, their accuracy is acceptable, though not as high as traditional
models. This is because the accuracy is dependent on
feature encoding which is not as well understood as
traditional classification.
B. Data Representation
Data points of HD computing correspond to hypervectors-vectors of bits, integers, real or complex numbers. These are roughly divided into two categories:
binary and non-binary. For non-binary hypervectors,
bipolar and integer hypervectors are more commonly
employed. Generally speaking, non-binary HD algorithms achieve higher accuracy, while the binary counterpart is more hardware-friendly and has higher efficiency (see also [11]).
C. Similarity Measurement
As shown in Table 2, two common similarity measurements are adopted in the existing HD algorithms, namely, cosine similarity and Hamming distance. Other similarity measures include dot product (e.g., in MAP) and
overlap (e.g., in BSDC).

For non-binary hypervectors, cosine similarity, defined by Eq. (1), is used to measure their similarity,
-focusing on the angle and ignoring the impact of the magnitude of hypervectors, where · denotes the magnitude. Unlike the inner product operation [12] of two vectors that affects magnitude and orientation, the cosine
similarity only depends on the orientation. In most HD
algorithms with non-binary hypervectors, cosine similarity is more often used than inner product. Moreover,
when cos (A, B) is close to 1, this implies an extremely
high level of similarity. For example, cos (A, B) = 1 indicates two hypervectors A and B are identical. When
they are at right angle, then cos (A, B) = 0, and the two
orthogonal vectors are considered dissimilar.
	

For binary hypervectors with dimensionality d, whose
components are either 0 or 1, normalized Hamming
distance calculated in Eq. (2) is used to measure their
similarity [8]. When the Hamming distance of two hypervectors is close to 0, then they are defined as similar.
For example, Ham (A, B ) = 0 indicates every single bit is
same at each position, and A and B are identical. When
Ham (A, B) = 0.5, A and B are orthogonal or dissimilar.
Ham (A, B) = 1 when A and B are diametrically opposed.
	

Table 2.
Similarity measurements in HD computing.
Measurement

Similar

Orthogonal

Hamming distance

0

0.5

Cosine similarity

1

0

90
d = 100
d = 500
d = 2,500
d = 10,000

80

Probability (%)

70
60
50
40
30
20
10
0

0

0.2
0.4
0.6
0.8
Normalized Hamming Distance

Figure 1. Orthogonality in high dimensions [1, 13, 14].
32 	

1

A·B (1)
A B

cos (A, B) =

d

Ham (A, B) = 1 / 1 A (i) ! B (i) (2)
d i= 1

One thing that should be emphasized is orthogonality in high dimensions. To put it simply, the randomly
generated hypervectors are nearly orthogonal to each
other when the dimensionality is in the thousands.
Take binary hypervectors as an example. Assume X
and Y are chosen independently and uniformly from
{0, 1} d and the probability p of any bit being 1 is 0.5.
Then Ham (X, Y ) is binomially distributed. Fig. 1 shows
the probability density function (PDF) of Ham (X, Y )
for 15,000 pairs of randomly selected binary vectors
with different dimensions d. As d increases, more vectors become orthogonal. Such orthogonality property
is of great interest because orthogonal hypervectors
are dissimilar. Moreover, operations performed on
these orthogonal hypervectors can form associations
or relations.
D. Data Transformation
Three types of operations, add-multiply-permute, are
employed in HD computing. The inverse operation for
multiplication is also referred to as release [14]. The
release operation is also used to denote inverse addition. Each operation processes and generates d-dimensional hypervectors. In the following, we -illustrate

IEEE CIRCUITS AND SYSTEMS MAGAZINE 		

SECOND QUARTER 2020
IEEE Circuits and Systems Magazine - Q2 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2020
