IEEE Circuits and Systems Magazine - Q2 2023 - 18
transformation is equivalent to a sequence of three separate
small convolutional kernels:
S
ZX
ZC
rw h
3,,
′′
rw h
4,,
=
=
tw h
,,
∑ ()
3
3 ,
s=1
R3
D
3== =
R4
j
i
, 4
∑∑∑
U
D
r rj ir wh ,
4 33 ji
Z
,,, ,, ,
r 11 1
A. Tensorizing Wx,
YZ ′′
r4 =1
′′ =∑U()
4
4,,
3×× and RW H respectively.
The rank- RR
4××′′,
34,
'
tr rw h ,
where Z and ′Z are the intermediate tensors of sizes
RW H
() parameters are very important in
the Tucker decomposition. They control the balance
between the model compression ratio and accuracy
loss. Kim et al. [19] applied global analytic solutions
for variational Bayesian matrix factorization (VBMF)
from [63] to determine the ranks of the Tucker decomposition.
VBMF can be used to find noise variance,
ranks and theoretical conditions for perfect rank recovery
[63].
IV. Tensorizing Recurrent Neural Networks
In this section, implementations of applying different
tensor decomposition methods to RNN models are described.
This section demonstrates the core steps of
NM⋅
∑⋅=k
d
kk
and y
Considering W is a 2-D matrix, it should be reshaped
since tensor decomposition methods are mainly applied
to high-order tensors. The input vector x
M and the weight matrix W MN
vector y∈R
and W∈ ,
=∏ i=1
R ×× ×× ××
mm nn nn
12
and Nnj
d
Mmi
d
∈R ×
∈∈
∈R , output
should be
N
tensorized into tensors XY 12
12 dd respectively, where
=∏ j=1 .
×× ×× ××
RR
nn nm mm
,
12 dd
B. Decomposing W
In general, the key idea of building tensor-decomposed
RNN is to transform the input-to-hidden weight matrices
in different tensor decomposition forms.
Tensor Train Decomposition Following the Tensor
Train Decomposition method, W can be decomposed
into [31]:
TTD WGGG() = 12 d.
Now instead of storing the full tensor W of size
=1 of size
, the set of low-rank core tensors {}G k
d
1nm . rr
kk−
1 ⋅ are stored.
Tensor Ring Decomposition Following
the Tensor Ring Decomposition
method, W can be decomposed
into [32]:
TRDWG G
G
() = ∑ ()
1
rr
,,
02 1
d−
()+12
01
dd+
,,
rm r
d
Grm r
d
()
−
dd
,,
21 0
For a d -order input and d -order
Figure 12. CP decomposed convolution layer in [15].
output, the weight tensor is decomposed
into 2d core tensors multiplied
one by one where each corresponding
to an input or an output
dimension.
Block Term Decomposition Following
the Block Term Decomposition
method, W can be decomposed
into [33]:
N
Figure 13. Tucker-2 decomposed convolution layer in [19].
18
IEEE CIRCUITS AND SYSTEMS MAGAZINE
BTD WG AA()=* **
=
∑
n 1
SECOND QUARTER 2023
nn d d
1
()
1
2
()
d
,
rn rr nr
d
01 11,, .
,, ()
−
dd d
rs sw h,
,,
four tensor-decomposed RNN models. 1) Transform W
and x into tensor representations, W and X; 2) Decompose
W into several lowrank tensors using different
tensor decomposition methods; 3) the original product
Wx⋅
is approximated by the tensor computation
between decomposed weight tensor W and the input
tensor X.
IEEE Circuits and Systems Magazine - Q2 2023
Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
Contents
IEEE Circuits and Systems Magazine - Q2 2023 - Cover1
IEEE Circuits and Systems Magazine - Q2 2023 - Cover2
IEEE Circuits and Systems Magazine - Q2 2023 - Contents
IEEE Circuits and Systems Magazine - Q2 2023 - 2
IEEE Circuits and Systems Magazine - Q2 2023 - 3
IEEE Circuits and Systems Magazine - Q2 2023 - 4
IEEE Circuits and Systems Magazine - Q2 2023 - 5
IEEE Circuits and Systems Magazine - Q2 2023 - 6
IEEE Circuits and Systems Magazine - Q2 2023 - 7
IEEE Circuits and Systems Magazine - Q2 2023 - 8
IEEE Circuits and Systems Magazine - Q2 2023 - 9
IEEE Circuits and Systems Magazine - Q2 2023 - 10
IEEE Circuits and Systems Magazine - Q2 2023 - 11
IEEE Circuits and Systems Magazine - Q2 2023 - 12
IEEE Circuits and Systems Magazine - Q2 2023 - 13
IEEE Circuits and Systems Magazine - Q2 2023 - 14
IEEE Circuits and Systems Magazine - Q2 2023 - 15
IEEE Circuits and Systems Magazine - Q2 2023 - 16
IEEE Circuits and Systems Magazine - Q2 2023 - 17
IEEE Circuits and Systems Magazine - Q2 2023 - 18
IEEE Circuits and Systems Magazine - Q2 2023 - 19
IEEE Circuits and Systems Magazine - Q2 2023 - 20
IEEE Circuits and Systems Magazine - Q2 2023 - 21
IEEE Circuits and Systems Magazine - Q2 2023 - 22
IEEE Circuits and Systems Magazine - Q2 2023 - 23
IEEE Circuits and Systems Magazine - Q2 2023 - 24
IEEE Circuits and Systems Magazine - Q2 2023 - 25
IEEE Circuits and Systems Magazine - Q2 2023 - 26
IEEE Circuits and Systems Magazine - Q2 2023 - 27
IEEE Circuits and Systems Magazine - Q2 2023 - 28
IEEE Circuits and Systems Magazine - Q2 2023 - 29
IEEE Circuits and Systems Magazine - Q2 2023 - 30
IEEE Circuits and Systems Magazine - Q2 2023 - 31
IEEE Circuits and Systems Magazine - Q2 2023 - 32
IEEE Circuits and Systems Magazine - Q2 2023 - 33
IEEE Circuits and Systems Magazine - Q2 2023 - 34
IEEE Circuits and Systems Magazine - Q2 2023 - 35
IEEE Circuits and Systems Magazine - Q2 2023 - 36
IEEE Circuits and Systems Magazine - Q2 2023 - 37
IEEE Circuits and Systems Magazine - Q2 2023 - 38
IEEE Circuits and Systems Magazine - Q2 2023 - 39
IEEE Circuits and Systems Magazine - Q2 2023 - 40
IEEE Circuits and Systems Magazine - Q2 2023 - 41
IEEE Circuits and Systems Magazine - Q2 2023 - 42
IEEE Circuits and Systems Magazine - Q2 2023 - 43
IEEE Circuits and Systems Magazine - Q2 2023 - 44
IEEE Circuits and Systems Magazine - Q2 2023 - 45
IEEE Circuits and Systems Magazine - Q2 2023 - 46
IEEE Circuits and Systems Magazine - Q2 2023 - 47
IEEE Circuits and Systems Magazine - Q2 2023 - 48
IEEE Circuits and Systems Magazine - Q2 2023 - 49
IEEE Circuits and Systems Magazine - Q2 2023 - 50
IEEE Circuits and Systems Magazine - Q2 2023 - 51
IEEE Circuits and Systems Magazine - Q2 2023 - 52
IEEE Circuits and Systems Magazine - Q2 2023 - 53
IEEE Circuits and Systems Magazine - Q2 2023 - 54
IEEE Circuits and Systems Magazine - Q2 2023 - 55
IEEE Circuits and Systems Magazine - Q2 2023 - 56
IEEE Circuits and Systems Magazine - Q2 2023 - 57
IEEE Circuits and Systems Magazine - Q2 2023 - 58
IEEE Circuits and Systems Magazine - Q2 2023 - 59
IEEE Circuits and Systems Magazine - Q2 2023 - 60
IEEE Circuits and Systems Magazine - Q2 2023 - 61
IEEE Circuits and Systems Magazine - Q2 2023 - 62
IEEE Circuits and Systems Magazine - Q2 2023 - 63
IEEE Circuits and Systems Magazine - Q2 2023 - 64
IEEE Circuits and Systems Magazine - Q2 2023 - 65
IEEE Circuits and Systems Magazine - Q2 2023 - 66
IEEE Circuits and Systems Magazine - Q2 2023 - 67
IEEE Circuits and Systems Magazine - Q2 2023 - 68
IEEE Circuits and Systems Magazine - Q2 2023 - 69
IEEE Circuits and Systems Magazine - Q2 2023 - 70
IEEE Circuits and Systems Magazine - Q2 2023 - 71
IEEE Circuits and Systems Magazine - Q2 2023 - 72
IEEE Circuits and Systems Magazine - Q2 2023 - Cover3
IEEE Circuits and Systems Magazine - Q2 2023 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com