IEEE Circuits and Systems Magazine - Q2 2023 - 15

where ww qj p and hh qi p . The
j =−
() +−
′ 1
i =−
() +−
′ 1
parameters p and q represent zero-padding size and
stride, respectively.
Fully connected layers apply a linear transformation
to an N -dimensional input vector x and compute a
M-dimensional output vector y :
yWxb
=+ .
(2)
where W and b represent the weight matrix and the
bias vector, respectively. In this section, we show how
these different tensor decomposition methods can be
applied to CNNs.
A. Tensor Train Decomposition
Tensor Train Decomposition can be applied to both the
convolutional layers and the fully connected layers of
CNNs. Let's first describe the implementation of TT-decomposition
on the fully connected layers.
Now consider the TT-representation of the weight ma=∏
k=1
trix W∈
R ×MN where Mmk
d
and νν ννtt ttd ()
W of size MN×
11 22
dd
and Nnk
d
=∏ k=1 . Define
two bijection mappings µµ µµtt ttd ()
() = () () ...12,, ,() that map the matrix
to a higher-order tensor W of size
() = () () ...12,, ,()
nm nm nm××... . The mapping µ ⋅() maps the row
index =...
12
=... of the matrix W into d -dimensional vec12
,,
,
kk and is indexed by
 

11
,
tt t
t
W µν µν
µν
,
() ()
,, ,M of the matrix W into a d -dimensional
vector-indices whose k -th dimensions are of
length mk. The mapping ν ⋅() maps the column index
tN
tor-indices whose k -th dimensions are of length nk.
Thus the k -th dimension of the reshaped d -dimensional
tensor W is of length nm
()
the tuple µν ()⋅kk⋅(),. Then the tensor W can be converted
using TT-decomposition:
W
()= ()
11 1 []
,, ,,()... dd
=G[]...() Gdd d tµν() ()
() () ()
() () ()
,.
The TT-format of the weight matrix transforms a d
dimensional tensor X (formed from the input vector x)
to the d -dimensional tensor Y (which can be used to
compute output vector y ). As illustrated in Fig. 10 from
[41], the linear transformation of a fully connected layer
can be computed in the TT-format:
YX
B
()... = []... [] ()...
1,,
iid
1111
jjd
,, ,, ,,
,,
+...(
...
1
∑GG
).
ij
iid
where B corresponds to the bias vector b in Eq. (2).
The ranks of the TT-format for the weight matrix depend
on the choice of the bijection mappings µ t() and ν t().
The computational complexity of the forward pass is
O(, ).
dr mM N
2 max{}
In machine learning, backpropagation is widely used
to train feed-forward neural networks based on the stochastic
gradient descent algorithm [24]. Backpropagation
computes the gradient of the loss-function L with
respect to all the parameters. Given the gradients with
respect to the layer's output ∂
∂
L
y , the backpropagation
applied to the fully connected layers computes the gradients
with respect to the input x, the weight matrix
W and bias vector b :
Figure 9. Original convolution layer with filter size of T × S
× D × D.
∂
∂
LL LL LL
x
=W

,, .
∂
∂
∂
=
∂
yW y
∂
∂
x
∂
∂
=
by
∂
∂
1
dd ddij jj
Figure 10. TT-format fully connected layer [41].
SECOND QUARTER 2023
IEEE CIRCUITS AND SYSTEMS MAGAZINE
15
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
