IEEE Circuits and Systems Magazine - Q2 2023 - 21

where the core tensor Gj is a diagonal tensor where j∈
1,
{}, h . SplitConcat () ⋅
following splitting. Matrices WW
qk,
parameter matrices that are shared.
B. Tensorized Embedding Layers
As shown in Fig. 14, the embedding layers convert
input words into vectors in NLP tasks. However, large
vocabulary results in enormous weight matrices, which
prevents their use in situations with constrained resources.
A parameter-efficient embedding layer, TTembedding,
was introduced [38]. The TTembedding
utilizes the Tensor Train decomposition by decomposing
the huge embedding matrix into a sequence of much
smaller 2-dimensional and 3-dimensional tensors. The
method is the same as applying the Tensor Train decomposition
to the fully connected layers of CNNs as
described before.
Let X∈R be the embedding matrix of size IJ× for
a vocabulary of size I and embedding dimension J.
Given factorizations of its dimensions IIk
=∏ k=1 , the embedding matrix can be reshaped
into a higher order tensor X∈
=∏ k=1 and
N
JJk
N
R ×× ×
IJ IJ
11 22  IJ . Then
NN
the high order tensor can be decomposed into a sequence
of smaller tensors Gk
=1
sequence Rk k{}N {}k=1
affect the compression ratio.
∈R 1− ×× ×kk kkRI JR N
. The
−1 represent the TT-ranks that directly
VI. Results
performs the concatenation
and Wv
are the
A. Convolutional Neural Networks
In this section, the implementations and results of applying
these tensor decomposition methods to some
classical CNN models are described. They are categorized
based on the tensor decomposition methods.
1) Tensor Train Decomposition: In [21], Tensor Train
decomposition was applied to the convolutional
layers and fully connected layers as described before.
The proposed method was tested on CIFAR-10
dataset from [67]. Two networks were built: one is
dominated by the convolutional layers which occupy
99 54
.% parameters of the network and the
other is dominated by the fully connected layers
which occupy 95 98
.% parameters of the network.
The first network consists of: conv with 64 output
channels; batch normalization; ReLU; conv with
64 output channels; batch normalization; ReLU;
maxpooling of size 33× with stride 2; conv with
128 output channels; batch normalization; ReLU;
max-pooling of size 33× with stride 2; conv with
128 output channels; batch normalization; ReLU;
conv with 128 output channels; avgpooling of size
44× ; FC of size 128 10×(), where 'conv' and 'FC'
stand for convolutional layer and fully connected
layer, respectively. All convolutional filters were
Figure 15. (left) Single-block attention using Tucker decomposition. (right) Multi-linear attention based on Block-Term tensor
decomposition [39].
SECOND QUARTER 2023
IEEE CIRCUITS AND SYSTEMS MAGAZINE
21

IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023

Contents
IEEE Circuits and Systems Magazine - Q2 2023 - Cover1
IEEE Circuits and Systems Magazine - Q2 2023 - Cover2
IEEE Circuits and Systems Magazine - Q2 2023 - Contents
IEEE Circuits and Systems Magazine - Q2 2023 - 2
IEEE Circuits and Systems Magazine - Q2 2023 - 3
IEEE Circuits and Systems Magazine - Q2 2023 - 4
IEEE Circuits and Systems Magazine - Q2 2023 - 5
IEEE Circuits and Systems Magazine - Q2 2023 - 6
IEEE Circuits and Systems Magazine - Q2 2023 - 7
IEEE Circuits and Systems Magazine - Q2 2023 - 8
IEEE Circuits and Systems Magazine - Q2 2023 - 9
IEEE Circuits and Systems Magazine - Q2 2023 - 10
IEEE Circuits and Systems Magazine - Q2 2023 - 11
IEEE Circuits and Systems Magazine - Q2 2023 - 12
IEEE Circuits and Systems Magazine - Q2 2023 - 13
IEEE Circuits and Systems Magazine - Q2 2023 - 14
IEEE Circuits and Systems Magazine - Q2 2023 - 15
IEEE Circuits and Systems Magazine - Q2 2023 - 16
IEEE Circuits and Systems Magazine - Q2 2023 - 17
IEEE Circuits and Systems Magazine - Q2 2023 - 18
IEEE Circuits and Systems Magazine - Q2 2023 - 19
IEEE Circuits and Systems Magazine - Q2 2023 - 20
IEEE Circuits and Systems Magazine - Q2 2023 - 21
IEEE Circuits and Systems Magazine - Q2 2023 - 22
IEEE Circuits and Systems Magazine - Q2 2023 - 23
IEEE Circuits and Systems Magazine - Q2 2023 - 24
IEEE Circuits and Systems Magazine - Q2 2023 - 25
IEEE Circuits and Systems Magazine - Q2 2023 - 26
IEEE Circuits and Systems Magazine - Q2 2023 - 27
IEEE Circuits and Systems Magazine - Q2 2023 - 28
IEEE Circuits and Systems Magazine - Q2 2023 - 29
IEEE Circuits and Systems Magazine - Q2 2023 - 30
IEEE Circuits and Systems Magazine - Q2 2023 - 31
IEEE Circuits and Systems Magazine - Q2 2023 - 32
IEEE Circuits and Systems Magazine - Q2 2023 - 33
IEEE Circuits and Systems Magazine - Q2 2023 - 34
IEEE Circuits and Systems Magazine - Q2 2023 - 35
IEEE Circuits and Systems Magazine - Q2 2023 - 36
IEEE Circuits and Systems Magazine - Q2 2023 - 37
IEEE Circuits and Systems Magazine - Q2 2023 - 38
IEEE Circuits and Systems Magazine - Q2 2023 - 39
IEEE Circuits and Systems Magazine - Q2 2023 - 40
IEEE Circuits and Systems Magazine - Q2 2023 - 41
IEEE Circuits and Systems Magazine - Q2 2023 - 42
IEEE Circuits and Systems Magazine - Q2 2023 - 43
IEEE Circuits and Systems Magazine - Q2 2023 - 44
IEEE Circuits and Systems Magazine - Q2 2023 - 45
IEEE Circuits and Systems Magazine - Q2 2023 - 46
IEEE Circuits and Systems Magazine - Q2 2023 - 47
IEEE Circuits and Systems Magazine - Q2 2023 - 48
IEEE Circuits and Systems Magazine - Q2 2023 - 49
IEEE Circuits and Systems Magazine - Q2 2023 - 50
IEEE Circuits and Systems Magazine - Q2 2023 - 51
IEEE Circuits and Systems Magazine - Q2 2023 - 52
IEEE Circuits and Systems Magazine - Q2 2023 - 53
IEEE Circuits and Systems Magazine - Q2 2023 - 54
IEEE Circuits and Systems Magazine - Q2 2023 - 55
IEEE Circuits and Systems Magazine - Q2 2023 - 56
IEEE Circuits and Systems Magazine - Q2 2023 - 57
IEEE Circuits and Systems Magazine - Q2 2023 - 58
IEEE Circuits and Systems Magazine - Q2 2023 - 59
IEEE Circuits and Systems Magazine - Q2 2023 - 60
IEEE Circuits and Systems Magazine - Q2 2023 - 61
IEEE Circuits and Systems Magazine - Q2 2023 - 62
IEEE Circuits and Systems Magazine - Q2 2023 - 63
IEEE Circuits and Systems Magazine - Q2 2023 - 64
IEEE Circuits and Systems Magazine - Q2 2023 - 65
IEEE Circuits and Systems Magazine - Q2 2023 - 66
IEEE Circuits and Systems Magazine - Q2 2023 - 67
IEEE Circuits and Systems Magazine - Q2 2023 - 68
IEEE Circuits and Systems Magazine - Q2 2023 - 69
IEEE Circuits and Systems Magazine - Q2 2023 - 70
IEEE Circuits and Systems Magazine - Q2 2023 - 71
IEEE Circuits and Systems Magazine - Q2 2023 - 72
IEEE Circuits and Systems Magazine - Q2 2023 - Cover3
IEEE Circuits and Systems Magazine - Q2 2023 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com