IEEE Circuits and Systems Magazine - Q2 2023 - 18

transformation is equivalent to a sequence of three separate
small convolutional kernels:
S
ZX
ZC
rw h
3,,
′′
rw h
4,,
=
=
tw h
,,
∑ ()
3
3 ,
s=1
R3
D
3== =
R4
j
i
, 4
∑∑∑
U
D
r rj ir wh ,
4 33 ji
Z
,,, ,, ,
r 11 1
A. Tensorizing Wx,
YZ ′′
r4 =1
′′ =∑U()
4
4,,
3×× and RW H respectively.
The rank- RR
4××′′,
34,
'
tr rw h ,
where Z and ′Z are the intermediate tensors of sizes
RW H
() parameters are very important in
the Tucker decomposition. They control the balance
between the model compression ratio and accuracy
loss. Kim et al. [19] applied global analytic solutions
for variational Bayesian matrix factorization (VBMF)
from [63] to determine the ranks of the Tucker decomposition.
VBMF can be used to find noise variance,
ranks and theoretical conditions for perfect rank recovery
[63].
IV. Tensorizing Recurrent Neural Networks
In this section, implementations of applying different
tensor decomposition methods to RNN models are described.
This section demonstrates the core steps of
NM⋅
∑⋅=k
d
kk
and y
Considering W is a 2-D matrix, it should be reshaped
since tensor decomposition methods are mainly applied
to high-order tensors. The input vector x
M and the weight matrix W MN
vector y∈R
and W∈  ,
=∏ i=1
R ×× ×× ××
mm nn nn
12
and Nnj
d
Mmi
d
∈R ×
∈∈
∈R , output
should be
N
tensorized into tensors XY 12
12 dd respectively, where
=∏ j=1 .
×× ×× ××
RR
nn nm mm
,
12 dd
B. Decomposing W
In general, the key idea of building tensor-decomposed
RNN is to transform the input-to-hidden weight matrices
in different tensor decomposition forms.
Tensor Train Decomposition Following the Tensor
Train Decomposition method, W can be decomposed
into [31]:
TTD WGGG() = 12 d.
Now instead of storing the full tensor W of size
=1 of size
, the set of low-rank core tensors {}G k
d
1nm . rr
kk−
1 ⋅ are stored.
Tensor Ring Decomposition Following
the Tensor Ring Decomposition
method, W can be decomposed
into [32]:
TRDWG G
G
() = ∑ ()
1
rr
,,
02 1
 d−
()+12
01
dd+
,,
rm r
d
Grm r
d
()
−
dd
,,
21 0
For a d -order input and d -order
Figure 12. CP decomposed convolution layer in [15].
output, the weight tensor is decomposed
into 2d core tensors multiplied
one by one where each corresponding
to an input or an output
dimension.
Block Term Decomposition Following
the Block Term Decomposition
method, W can be decomposed
into [33]:
N
Figure 13. Tucker-2 decomposed convolution layer in [19].
18
IEEE CIRCUITS AND SYSTEMS MAGAZINE
BTD WG AA()=* **
=
∑
n 1
SECOND QUARTER 2023
nn d d
1
()
1
2
()
d
,
rn rr nr
d
01 11,, .
,,  ()
−
dd d
rs sw h,
,,
four tensor-decomposed RNN models. 1) Transform W
and x into tensor representations, W and X; 2) Decompose
W into several lowrank tensors using different
tensor decomposition methods; 3) the original product
Wx⋅
is approximated by the tensor computation
between decomposed weight tensor W and the input
tensor X.
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
