IEEE Circuits and Systems Magazine - Q2 2023 - 14

CP-rank. RR Rd12
,, , are the Tucker-rank where d is
the Core-order. Figure 7 shows the block term decomposition
for a 3rd -order tensor.
G. Hierarchical Tucker Decomposition
Hierarchical Tucker decomposition has multiple hierarchical
levels based on the order of the tensor. From
top to bottom in a binary tree, a hierarchical tucker
decomposed tensor can be recursively decomposed
into intermediate tensors, referred as frames, as shown
in Fig. 8. Each frame is a unique node where each node
is associated with a dimension set. Given a d -dimensional
target tensor X∈
R ×××
II Id12 
a root node associated with Dd={}
rIµν×× ×I
12
∈R⊊
,
the left and right child nodes of the s -associated node
where µs
= mi ()n
s
((G∈
R ××
rr rss s12
and νs = x. A non-leaf frame
ma ()
s
Us can be recursively decomposed to a left child frame
() Us1
, a right child frame Us2() and a transfer tensor
)) as follows [34]:
UG UUss ss=× ×
1
2
12,
1
2
where ×1
2 denotes the tensor contraction that can be executed
between two tensors with at least one matched dimension.
For example, given two tensors A∈
R ××
B∈R ××
, a binary tree with
,, , can be built
where XU= D is the root frame. For each non-leaf frame
Us
s ss sD is associated with the node corresponding
to Us and ss s12, ⊊ are associated with
lm m12
nn l12
and
where the third dimension of A matches
×× ×
AB ,, ,,,, ,AB×=∑ =
1
3
ii jj αα jjα 12 . The origil
1
212
1
ii
12
the first dimension of B, a tensor of size nn mm1 212
can be computed by using the tensor contraction operation
as ()
nal IId
1×× -order tensor XU= D can be recursively
decomposed into a combination of the 2nd -order leaf
frames and the 3rd -order transfer
tensors by performing the hierarchical
tucker decomposition from
top to bottom of the binary tree.
The parameter rs is defined as hierarchical
rank.
III. Tensorizing Convolutional
Neural Networks
Figure 7. Block Term decomposition for a 3rd-order tensor. The tensor can be approximated
by N Tucker decompositions where N is the CP-rank; R1, R2, R3 are the
Tucker-rank; d is the Core-order [33].
CNNs have two main parts: convolutional
layers and fully connected
layers. In general, convolutional
layers in CNNs map a third-order
input tensor X of size SW H××
into a third-order output tensor
Y of size TW H××′′ with a 4th
order kernel tensor K of size
TS DD×× × , where DD×
represents
the filter size, S and T
represent the number of input and
output channels, respectively. The
typical convolutional filter sizes
are small, e.g.,
33 77 , com××,
pared
to the numbers of input S()
and output T() channels. Each
convolutional layer may have hundreds
or thousands of filters which
are suited for tensor decomposition
methods. As shown in Fig. 9,
the output tensor Y can be computed
as:
Figure 8. Hierarchical Tucker decomposition for a 4th-order tensor. It is a binary tree
with root D = {1, 2, 3, 4} where the dashed boxes represent the nodes. Node {1} is a
leaf node whose parent and sibling are node {1, 2} and node {2}, respectively [34].
14
IEEE CIRCUITS AND SYSTEMS MAGAZINE
S
tw h
,,
D
j
i
D
YK Xts ji sw hji
′′ =
∑∑∑
s 11 1== =
SECOND QUARTER 2023
,, ,, ,
,
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
