IEEE Circuits and Systems Magazine - Q2 2023 - 23

Figure 16. CP-TPM method in AlexNet [15]. The black solid arrow represents the connection between layers. The red dotted line
stands for the decomposition process and the black dotted line means that the weights are taken from the previous iteration. The
purple block arrow stands for fine-tuning by backpropagation to all the layers. First, Conv 1 is decomposed into 2 layers while
the others remain the same. Then, fine-tuning is performed on the whole network. Afterward, Conv2 is decomposed into 3 layers
and then fine-tuning is performed on the whole network followed. The process repeats until all the layers are decomposed and
fine-tuned.
The next iteration approximates the residual tensor
KK K
residual −
.
residual =− 1 by minimizing KK2 2 This
continues until KR is found. More details can be
found in [69]. It is the first time that CP-based decomposition
is applied to the whole convolutional layer in
[15]. AlexNet was used here. The authors overcome
the instability of CP decomposition by fine-tuning after
each layer's decomposition. The fully connected
layers were decomposed using SVD as described before.
Figure 16 from [15] shows that decomposition
and finetuning are performed iteratively from Conv1
to FC8. Black solid arrows represent the connection
between each layer. Red dotted lines represent the
decomposition processes. Black dotted lines show
that the weights do not change from the previous
iteration. Purple block arrows represent fine-tuning
by backpropagation to the whole network. The rank
of each layer was set to be proportional to its sensitivity
which is defined as loss/total_loss. This method
achieved 698. × parameter reduction and 353. × running
time reduction with the expense of 142.% accuracy
loss.
3) Tucker Decomposition:
In [19], one-shot Tucker
Decomposition on the whole network consists
SECOND QUARTER 2023
of three steps: rank selection using VBMF, Tucker
decomposition on each layer's kernel tensor
and one-shot fine-tuning the whole network
with standard back-propagation. Figure 17 from
[19] shows the whole scheme. The accuracy significantly
dropped after step two but recovered
quickly in one epoch. Four representative CNNs,
AlexNet, VGG-S, GoogLeNet and VGG-1 6
were compressed using Tucker decomposition in
[19]. For GoogLeNet, only the 33× convolution
kernel was compressed in the case of inception
module. For VGG-16, only the convolutional
layers were compressed. This method achieved
546. × / 2.67× (AlexNet), 7.40× / 4.80× (VGGS),
1.28× / 2.06× (GoogLeNet) and 1.09× / 4.93×
(VGG-16) reductions in total weights and FLOPs,
respectively.
B. Recurrent Neural Networks
In this section, the results of applying different tensor
decomposition methods to RNN models are described.
To better understand the impact of different tensor
decomposition methods on RNNs, the time complexity
and space complexity of different RNN models are
IEEE CIRCUITS AND SYSTEMS MAGAZINE
23

IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023