IEEE Circuits and Systems Magazine - Q2 2023 - 24

Figure 17. One-shot whole network compression based on Tucker decomposition [19]. Tucker-2 decomposition is applied from
the second convolutional layer to the first fully connected layers while Tucker-1 decomposition is applied to the other layers.
Table 4.
Comparison of time complexity and space complexity
of vanilla RNN, Tensor Train decomposed RNN (TTRNN)
[31], Tensor Ring decomposed RNN (TR-RNN)
[32], Block-Term decomposed RNN (BT-RNN) [33],
and Hierarchical Tucker decomposed RNN (HT-RNN)
[34]. The weight matrix's shape is M × N. The input
and hidden tensors' shapes are N = n1 × · · · × nd and
M = m1 × · · · × md, respectively. Here, n = max(nk)
for k ∈ [1, d] and m = max(nk) for k ∈ [1, d]. All
tensor decomposed models are set in the same rank
R. C is the CP-rank defined in BT-RNN model.
Method
RNN forward
RNN backward
TT-RNN forward
TT-RNN
backward
TR-RNN forward
TR-RNN
backward
BT-RNN forward
BT-RNN
backward
HT-RNN forward
HT-RNN
backward
Time
O(NM)
O(NM)
O(dmR2N)
O(d2mR4N)
O(dR3N +
dR3M)
O(d2R5N +
nd2R5M)
O(dmRdNC)
O(d2mRdNC)
O(dmR2N +
dR3N)
O(d2mr5N +
d2R6N)
Space
O(NM)
O(NM)
O(RM)
O(R3M)
O(dmnR2)
O(dmnR2)
O(RdM)
O(RdM)
O(dmnR +
dR3)
O(dmnR +
dR3)
summarized in Table 4. Notice that HT-RNN has the lowest
space complexity and TT-RNN has the lowest time
complexity.
The UCF11 YouTube Action dataset [70] contains
1600 video clips of a resolution 320 240×
24
IEEE CIRCUITS AND SYSTEMS MAGAZINE
etc.). Each category contains 25 video groups, within
each contains more than 4 clips. Table 5 summarizes
the performance of different tensordecomposed LSTM
models on UCF11 dataset. Notice that HT-LSTM requires
at least 138. × fewer parameters with at least
03
.% increase in accuracy compared to the other tensordecomposed
LSTM models.
C. Transformers
In this section, the results of applying different tensor
decomposition methods to Transformers are described.
1) Multi-Linear Attention: The proposed attention
method was tested on three language modeling
tasks (PTB, WikiText103 and One-billion) and a
neural machine translation task (WMT-2016 English-German)
[39].
Language modeling is the task of predicting the next
, falling into
11 action categories (e.g., basketball, biking, diving,
word or character in a document. It can be used to generate
text or further fine-tuned to solve different NLP
SECOND QUARTER 2023
Table 5.
Comparison of different tensor-decomposed
LSTM models on UCF11 dataset. CR stands for
compression ratios.
Method
LSTM
TT-LSTM
[31]
TRLSTM
[32]
BT-LSTM
[33]
HTLSTM
[34]
CR
1
17554×
34193×
17414×
47375×
#
of Param.
59M
3360
1725
3387
1245
Accuracy
69.7 %
79.6 %
86.9 %
85.3 %
87.3 %

IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023

Contents
IEEE Circuits and Systems Magazine - Q2 2023 - Cover1
IEEE Circuits and Systems Magazine - Q2 2023 - Cover2
IEEE Circuits and Systems Magazine - Q2 2023 - Contents
IEEE Circuits and Systems Magazine - Q2 2023 - 2
IEEE Circuits and Systems Magazine - Q2 2023 - 3
IEEE Circuits and Systems Magazine - Q2 2023 - 4
IEEE Circuits and Systems Magazine - Q2 2023 - 5
IEEE Circuits and Systems Magazine - Q2 2023 - 6
IEEE Circuits and Systems Magazine - Q2 2023 - 7
IEEE Circuits and Systems Magazine - Q2 2023 - 8
IEEE Circuits and Systems Magazine - Q2 2023 - 9
IEEE Circuits and Systems Magazine - Q2 2023 - 10
IEEE Circuits and Systems Magazine - Q2 2023 - 11
IEEE Circuits and Systems Magazine - Q2 2023 - 12
IEEE Circuits and Systems Magazine - Q2 2023 - 13
IEEE Circuits and Systems Magazine - Q2 2023 - 14
IEEE Circuits and Systems Magazine - Q2 2023 - 15
IEEE Circuits and Systems Magazine - Q2 2023 - 16
IEEE Circuits and Systems Magazine - Q2 2023 - 17
IEEE Circuits and Systems Magazine - Q2 2023 - 18
IEEE Circuits and Systems Magazine - Q2 2023 - 19
IEEE Circuits and Systems Magazine - Q2 2023 - 20
IEEE Circuits and Systems Magazine - Q2 2023 - 21
IEEE Circuits and Systems Magazine - Q2 2023 - 22
IEEE Circuits and Systems Magazine - Q2 2023 - 23
IEEE Circuits and Systems Magazine - Q2 2023 - 24
IEEE Circuits and Systems Magazine - Q2 2023 - 25
IEEE Circuits and Systems Magazine - Q2 2023 - 26
IEEE Circuits and Systems Magazine - Q2 2023 - 27
IEEE Circuits and Systems Magazine - Q2 2023 - 28
IEEE Circuits and Systems Magazine - Q2 2023 - 29
IEEE Circuits and Systems Magazine - Q2 2023 - 30
IEEE Circuits and Systems Magazine - Q2 2023 - 31
IEEE Circuits and Systems Magazine - Q2 2023 - 32
IEEE Circuits and Systems Magazine - Q2 2023 - 33
IEEE Circuits and Systems Magazine - Q2 2023 - 34
IEEE Circuits and Systems Magazine - Q2 2023 - 35
IEEE Circuits and Systems Magazine - Q2 2023 - 36
IEEE Circuits and Systems Magazine - Q2 2023 - 37
IEEE Circuits and Systems Magazine - Q2 2023 - 38
IEEE Circuits and Systems Magazine - Q2 2023 - 39
IEEE Circuits and Systems Magazine - Q2 2023 - 40
IEEE Circuits and Systems Magazine - Q2 2023 - 41
IEEE Circuits and Systems Magazine - Q2 2023 - 42
IEEE Circuits and Systems Magazine - Q2 2023 - 43
IEEE Circuits and Systems Magazine - Q2 2023 - 44
IEEE Circuits and Systems Magazine - Q2 2023 - 45
IEEE Circuits and Systems Magazine - Q2 2023 - 46
IEEE Circuits and Systems Magazine - Q2 2023 - 47
IEEE Circuits and Systems Magazine - Q2 2023 - 48
IEEE Circuits and Systems Magazine - Q2 2023 - 49
IEEE Circuits and Systems Magazine - Q2 2023 - 50
IEEE Circuits and Systems Magazine - Q2 2023 - 51
IEEE Circuits and Systems Magazine - Q2 2023 - 52
IEEE Circuits and Systems Magazine - Q2 2023 - 53
IEEE Circuits and Systems Magazine - Q2 2023 - 54
IEEE Circuits and Systems Magazine - Q2 2023 - 55
IEEE Circuits and Systems Magazine - Q2 2023 - 56
IEEE Circuits and Systems Magazine - Q2 2023 - 57
IEEE Circuits and Systems Magazine - Q2 2023 - 58
IEEE Circuits and Systems Magazine - Q2 2023 - 59
IEEE Circuits and Systems Magazine - Q2 2023 - 60
IEEE Circuits and Systems Magazine - Q2 2023 - 61
IEEE Circuits and Systems Magazine - Q2 2023 - 62
IEEE Circuits and Systems Magazine - Q2 2023 - 63
IEEE Circuits and Systems Magazine - Q2 2023 - 64
IEEE Circuits and Systems Magazine - Q2 2023 - 65
IEEE Circuits and Systems Magazine - Q2 2023 - 66
IEEE Circuits and Systems Magazine - Q2 2023 - 67
IEEE Circuits and Systems Magazine - Q2 2023 - 68
IEEE Circuits and Systems Magazine - Q2 2023 - 69
IEEE Circuits and Systems Magazine - Q2 2023 - 70
IEEE Circuits and Systems Magazine - Q2 2023 - 71
IEEE Circuits and Systems Magazine - Q2 2023 - 72
IEEE Circuits and Systems Magazine - Q2 2023 - Cover3
IEEE Circuits and Systems Magazine - Q2 2023 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com