IEEE Circuits and Systems Magazine - Q2 2023 - 10

Like RNNs, transformers are designed to process sequential
input. However, transformers process the entire
input all at once. For example, the transformers
can process the whole natural language sentence at
a time while the RNNs have to process word by word.
The training parallelization allows transformers to be
trained on larger datasets. This has led to the success of
pre-trained systems such as BERT [35], GPT [36], and T5
[37]. However, the large model size of the Transformer
based model may cause problems in training and inference
under resourcelimited environments. Tensorized
embedding (TE) utilized the Tensor Train decomposition
to compress the embedding layers of Transformers
[38]. A novel self-attention model with Block-Term
Decomposition was proposed to compress the attention
layers of Transformers [39]. This method can not only
largely compress the model size but also achieve performance
improvement.
II. Tensor Decomposition Methods
A tensor decomposition is any scheme for compressing
a tensor into a sequence of other, often simpler tensors.
In this section, we review some tensor decomposition
methods that are commonly used to compress deep
learning models.
A. Truncated Singular Value Decomposition
Given a matrix W∈
sition (SVD) of the matrix is defined as:
WUSV=
,
where US MN
××
RR
MM
,
∈∈ and V∈R ×NN. S is the diagonal
matrix with all the singular values on the diagonal.
U and V are the corresponding orthogonal matrices.
If the singular values of W decay fast, then the weight
R ×MN , the singular value decompomatrix
()W can be approximated by keeping only the
K largest entries of S:
  
WUSV=
whereUS, ∈∈ KK

××
RR
MK
I∈
IW
,
and V  ∈R ×KN. Then for any
R ×TM, the SVD approximation error satisfies:
 
IW −≤ I+FK F,
s 1
where ⋅ F denotes its Frobenius norm. Notice that
the approximation error IW IW−
by sK+1, the ()K th
F
+1
in O() which is much smaller than
O()
TMKTKTKN++2
TMN for a sufficiently small K.
B. Tensor Train Decomposition
As defined in [40], the Tensor Train (TT) Decomposition
of a tensor A∈
R ×× ×nn nd12 
placed by a set of matrices Gkk
=... =...12
jnkdkk
12
,, ,, ,, , 12
,,,....
11 22
[]∈R ×−1
,, ,... d and rrd
()= [] []...GG []G
12 dd d
{}={} 1
n
[] k
j
in TT-format, only ∑ =−k
d
k
k
d
=1
has ∏ =k
d
j rr
kk
each of the tensor elements can be computed as:
A jj jj jj
The sequence rk k
{}d
=1 is called TT-rank of the TTrepresentation
of A. The collections of the matrices
Gkk j
nr rkk k
to represent a tensor A∈R ×× ×nn nd12 
{}d
are defined as TT-cores [40]. Notice that
11 parameters are required
which originally
1 nk elements. The trade-off between the model
compression ratio and the reconstruction accuracy is
controlled by the TT-ranks ()rk k
=1 . The smaller the TTranks,
the higher the model compression ratio TT-format
can achieve. Another advantage of the TT-decomposition
is that basic linear algebra
operations can be applied
to the TT-format tensors efficiently
[40]. Figure 2 from
[41] shows that a third tensor
A∈
R ××
()=1
3
34 5
can be represented
by three TT-cores
{} Gd d
with 32 parameFigure
2. Tensor Train Decomposition of a third-order tensor [41]. The original tensor has the
dimensions 3 × 4 × 5. After decomposition, the dimensions of the three TT-cores are 1 × 3 ×
2, 2 × 4 × 2, and 2 × 5 × 1, respectively. The number of parameters needed to represent the
original tensor is reduced from 60 to 32.
10
IEEE CIRCUITS AND SYSTEMS MAGAZINE
ters in TT-format. Thus, the
number of parameters needed
to represent the original
tensor is reduced from 60
to 32.
Tensor Train Decomposition
utilizes two key
ideas: recursively applying
SECOND QUARTER 2023
largest singular value, or put
another way, the decay along the diagonal of S. Considering
the computation cost, IW
can be computed
is controlled
can be rewhere
==1. Then
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
