IEEE Circuits and Systems Magazine - Q2 2023 - 17

All these models are trained end-to-end using backpropagation instead of being
obtained by applying tensor decomposition methods to the pre-trained
standard models. It is possible for them to achieve higher accuracy
than the standard models with fewer parameters.
First, the TT-convolution layer reshapes the
input tensor into a d +()2 -dimensional tensor
X∈R ×× ××SS WHd
1 
into the output tensor Y∈R ×× ×− +()×− +()TT WD HD
1 11
ing the following equation:
Y
()...1,, ,,dtt yx
D
D
=... +− +−
i 11 1,,== ... d
X ss jy ix
1,, ,,
d
j ss
× [] []... []GGyD xt st sdd d
0 +−
()
∑∑ ∑
G 111 11 1,, ,.
()
11
where Zrw h,,
While training the network, the stochastic gradient
is applied to each element of the TT-cores with momentum
[21].
B. Canonical Polyadic Decomposition
In CNNs, convolution layers map an input tensor of size
SW H×× into an output tensor of size TW H××′′ using
a kernel tensor of size TS DD×× × where S and T represent
different input and output channels, respectively.
Now consider how to approximate the kernel tensor K
with rank- R CP-decomposition. The number of the parameters
needed to represent the kernel tensor of size
TS DD×× × after the decomposition is RD TS
()2 ++
since the 4 dimensional kernels are reshaped to 3-dimensional
kernels of size TS D×× 2 as the filter size D is relatively
small (e.g., 33× or 55× ). The rank- RCP format of
the reshaped kernel tensor can be represented as [15]:
R
KU ,
,,,, ,,
r=1
where UUtr
() ()
rs Urj i
,
,,
,, ,
12 3
ts ji =∑ () () ()
rs rj i tr
12 3
UU .
() are the three tensors of sizes
RS× , RD D×× and TR× , respectively. Then the approximate
transformation of the convolution from the
input tensor X to the output tensor Y is given by:
R
YU XUU1
r=1
tw h
,,
′′ =∑∑∑∑ ()
()
3
tr,




j 11
==
i
D
D
()
,,
2
rj i




s=1
S
rs swj hi
,, ,








.
R3
As shown in Fig. 12 from [15], this transformation is
equivalent to a sequence of three separate small convolutional
kernels in a row:
SECOND QUARTER 2023
R4
KC UUrr ji rs rt
r4
ts ji =∑∑
34
,,,, ,,43 34
r 113==
() ()
,, ,
where C represents the core tensor of size
RR DD43
×× × . As shown in Fig. 13 from [19],
and Zrw h,,′′
'
are intermediate feature tensors
of sizes RW H×× and RW H××′′, respectively.
After this replacement, the entire network can be
trained using a standard backpropagation process.
Ranks play an important role in CP decomposition. Unfortunately,
no polynomial time algorithm for determining
the rank of a tensor exists [46], [48]. Therefore, most
algorithms approximate the tensor with different ranks
until a " good " rank is found. Astrid and Lee [15] applied
rank-5 CP decomposition first and then fine-tuned the
whole network to balance the accuracy loss and the
number of ranks. Phan et al. [56] applied a heuristic binary
search to find the smallest rank such that the accuracy
drop after every single layer is acceptable.
C. Tucker Decomposition
The 4th -order kernel tensor K can be decomposed by
the rank- RR RR12 34,, ,() Tucker decomposition as:
KG3
r2
ts ji =∑ ∑∑ ∑R4
R1
R2
,,,,,,
r 11 11 rr rr ri
1== == 43 21 1,
R
r3
r4
'
UUUU U
rj rs rt23 4
()
1
() () ()
23 4
,,, ,
where G ′ is the core tensor of size RR RR43 21×× × and
() () (),,
UU U12 3 and U 4() are the factor matrices of sizes
of sizes RD RD RS12 3×× ×,, and RT4 × , respectively.
As mentioned before, the filter size D is relatively
small compared to the number of input and output channels.
Mode1 and mode-2 which are associated with the
filter sizes don't need to be decomposed. Under this
condition, the kernel tensor can be decomposed by the
Tucker-2 decomposition as follows:
and then transforms this tensor
d
usS
ZX
ZU
Z
rw h
,,
'
′′
rw h
,,
=
=
tw h
,,
∑ ()
1
s=1
D
==
i
D
∑∑ ()
2
j 11
R
YZ .
r=1
′′ =∑ ()
U 3 '
′′
tr rw h
, ,,
rj i t wh,wji,
,,
,
Urs sw h,,, ,
this
IEEE CIRCUITS AND SYSTEMS MAGAZINE
17
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
