IEEE Circuits and Systems Magazine - Q2 2023 - 9

A tensor decomposition is any scheme for compressing a
tensor into a sequence of other, often simpler tensors.
I. Introduction
T
ensors are multidimensional arrays indexed by
three or more indices. An Nth -order tensor is
the tensor product of N vector spaces. Thirdorder
tensors have three indices as shown in Fig. 1. In
special cases, first-order tensors represent vectors, and
second-order tensors represent matrices.
Convolutional neural networks (CNNs) have outperformed
traditional techniques for image recognition
tasks. In 2012, AlexNet [1] achieved about 80% top-5
accuracy on the ImageNet dataset [2]. Furthermore,
subsequently VGG [3] and GoogleNet [4] achieved
about 90% top-5 accuracy with the same dataset. On
the ImageNet dataset, ResNet [5] with a depth of up to
152 layers achieved 357.% top-5 error. Executing CNNs
for computer vision tasks on mobile devices is gaining
more and more attention. Common methods to reduce
the size of CNNs include: sparsification [6], [7], [8], [9],
quantization [1], [10], structural pruning [11], [12], [13],
[14], and low-rank approximation [15], [16], [17], [18],
[19], [20], [21], [22].
The use of low-rank approximations is inspired by
[23] which showed that the neural network parameters
are highly redundant. The authors in this article could
predict more than 95% of the weights of the network
which indicates that the parameters of the CNNs are
highly over-parametrized. Various low-rank tensor/
matrix decompositions can be directly applied to the
weights of convolutional and fully connected layers. In
this article, we review Canonical Polyadic decomposition
(CPD) [15], [16], [17], Tucker decomposition [18], [19]
and Tensor Train decomposition [20], [21] approaches
to reduce model parameters of CNNs. The decomposed
layers are replaced by a sequence of new layers with significantly
fewer parameters.
Recurrent Neural Networks (RNNs) have shown
promising success in sequence modeling tasks due to
their ability in capturing temporal dependencies from
input sequences [24], [25]. Their advanced variant, the
Long Short-Term Memory (LSTM), introduces a number
of gates and passes information with element-wise
operations to address the gradient vanishing issue in
vanilla RNNs [26]. These neural network architectures
have shown promising performances in Natural Language
Processing (NLP) [27], speech recognition [28],
and computer vision (CV) [29]. The reader is referred
to [30] for a review of various brain-inspired computing
models.
Despite the success, RNNs and LSTMs suffer from a
huge number of parameters which make the models notoriously
difficult to train and susceptible to overfitting.
In order to circumvent this problem, current approaches
often involve exploring the low-rank structures of
the weight matrices. Inspired by the implementation of
tensor decomposition methods in CNNs [20], various
tensor decomposition methods have been applied in
RNNs and LSTMs, including Tensor Train Decomposition
[31], Tensor Ring Decomposition [32], BlockTerm
Decomposition [33], and Hierarchical Tucker Decomposition
[34]. These tensor-decomposed models can
maintain high performance with orders-of-magnitude
fewer parameters compared to the large-size vanilla
RNNs/LSTMs.
Transformer is a deep learning model that is based
on the mechanism of self-attention by weighting the significance
of each part of the input data differentially. It
has led to breakthroughs in the fields of NLP and CV.
Figure 1. A third-order tensor: χ ∈ RI×J×K.
The authors are with the Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455 USA (e-mail: parhi@
umn.edu).
SECOND QUARTER 2023
IEEE CIRCUITS AND SYSTEMS MAGAZINE
9
IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023
