IEEE Circuits and Systems Magazine - Q2 2023 - 20

an output sequence utilizing the contextual information.
Both encoder and decoder layers make use of an attention
mechanism. The attention function is called " Scaled DotProduct
Attention " . It computes an output as a weighted
sum of the values, with each value's weight determined
by the compatibility function of the query with the corresponding
key. In practice, the attention function is computed
on a set of queries simultaneously by packing queries,
keys and values into matrices QK,
and V, respectively.
Then the matrix of outputs can be computed as [65]:
Attentionsoftmax
()=
QK V
,,



QK
T
d V,


where d is the number of columns of Q and K. Instead
of performing a single attention function, it is beneficial
to linearly project the queries, keys and values h times
using distinct linear projections. The attention function
is then applied concurrently to each of these projected
versions of queries, keys, and values. The final values
of the attention function with different inputs are then
concatenated and projected to generate the final values.
Multi-head attention allows the model to learn from different
representations at different positions. It can be
expressed as follows [65]:
MultiHead
wherehead Attenti
QKVWh
()= ()
=
,,
i ()i
Q
Concat head ,,1  head
K
where matrices WWi
Q
,
dd
i
model × and WOhdd
∈R × model
K and Wi
.
ono QW KW VWV
i ,, ,
i
V have the same size of
Figure 15 (left) shows the schematic diagram of
Single-block attention based on the Tucker decomposition.
The query, key and value can be mapped into
three-factor matrices (,
QK and V) where each factor
matrix is composed of three groups of orthogonal basis
vectors. Then a new attention can be constructed
by initializing a 3rd-order diagonal tensor G as shown
in Fig. 15 (left) where RN,
O
and d represent the rank
of the tensor, the length of the input sequence and the
dimension of the matrix, respectively. The Tuckerdecomposition
inspired Single-block attention can be
computed as [39]:
I
j
J
M
AttenTDGG 
== =11 1
() ijmi jm
=
;,,,QKVQ KV
i
∑∑∑
m
where G is the trainable core tensor and  represents
the outer product. QK
vectors of matrices QK,
ij,
eters ij, and m are the indexes of the core tensor. In
practice, the core tensor G can be defined as [39]:
Gijk =
{
rand 01 ij m
() ==
,
otherwise
where rand 01,() represent a random function. Then the
Scaled Dot-Product attention can be computed by summing
over the output of the Single-block attention function
according to the second index as follows [39]:
N
AttentionAttenTD G
(, ,)QK ,,,VQKV ij m
j=1
im =∑
(; ,, ),
where ij, and m represent the indices of the output of
the single-block attention.
In order to compress the multi-head function,
multi-linear attention was proposed utilizing the idea
of parameter sharing. As shown in Fig. 15 (right), a set
of linear projections map queries, keys and values to
three matrices that can be composed of basis vectors.
Then the multi-linear attention can be computed as
follows [39]:
MultiLinear G;, ,QK V
Figure 14. Model architecture of the Transformer [65].
20
IEEE CIRCUITS AND SYSTEMS MAGAZINE
()=∗ ++ 
h
whereT n,
QWqk v
SplitConcat TT WO
jjj = Atte ()TD G ;, ,KW VW
 ()

1
1  h


SECOND QUARTER 2023
and Vk represent the column
and V, respectively. Param
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
