IEEE Circuits and Systems Magazine - Q2 2023 - 8

Feature
IMAGE LICENSED BY INGRAM PUBLISHING
Tensor Decomposition
for Model Reduction in
Neural Networks: A Review
Xingyi Liu, Graduate Student Member, IEEE and Keshab K. Parhi, Fellow, IEEE
Abstract
Modern neural networks have revolutionized the fields of computer
vision (CV) and Natural Language Processing (NLP).
They are widely used for solving complex CV tasks and NLP
tasks such as image classification, image generation, and machine
translation. Most state-of-the-art neural networks are
over-parameterized and require a high computational cost. One
straightforward solution is to replace the layers of the networks
with their low-rank tensor approximations using different tensor
decomposition methods. This article reviews six tensor decomposition
methods and illustrates their ability to compress model
parameters of convolutional neural networks (CNNs), recurrent
Digital Object Identifier 10.1109/MCAS.2023.3267921
Date of current version: 20 July 2023
8
IEEE CIRCUITS AND SYSTEMS MAGAZINE
1531-636X/23©2023IEEE
SECOND QUARTER 2023
neural networks (RNNs) and Transformers. The accuracy of
some compressed models can be higher than the original versions.
Evaluations indicate that
tensor decompositions can
achieve significant reductions in model size, run-time and energy
consumption, and are well suited for implementing neural
networks on edge devices.
Index Terms-Tensor decomposition, convolution neural network
acceleration, recurrent neural network acceleration, transformer
acceleration, canonical polyadic decomposition, Tucker decomposition,
tensor train decomposition, tensor ring decomposition,
block-term decomposition, hierarchical Tucker decomposition,
model compression.

IEEE Circuits and Systems Magazine - Q2 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2023

Contents
IEEE Circuits and Systems Magazine - Q2 2023 - Cover1
IEEE Circuits and Systems Magazine - Q2 2023 - Cover2
IEEE Circuits and Systems Magazine - Q2 2023 - Contents
IEEE Circuits and Systems Magazine - Q2 2023 - 2
IEEE Circuits and Systems Magazine - Q2 2023 - 3
IEEE Circuits and Systems Magazine - Q2 2023 - 4
IEEE Circuits and Systems Magazine - Q2 2023 - 5
IEEE Circuits and Systems Magazine - Q2 2023 - 6
IEEE Circuits and Systems Magazine - Q2 2023 - 7
IEEE Circuits and Systems Magazine - Q2 2023 - 8
IEEE Circuits and Systems Magazine - Q2 2023 - 9
IEEE Circuits and Systems Magazine - Q2 2023 - 10
IEEE Circuits and Systems Magazine - Q2 2023 - 11
IEEE Circuits and Systems Magazine - Q2 2023 - 12
IEEE Circuits and Systems Magazine - Q2 2023 - 13
IEEE Circuits and Systems Magazine - Q2 2023 - 14
IEEE Circuits and Systems Magazine - Q2 2023 - 15
IEEE Circuits and Systems Magazine - Q2 2023 - 16
IEEE Circuits and Systems Magazine - Q2 2023 - 17
IEEE Circuits and Systems Magazine - Q2 2023 - 18
IEEE Circuits and Systems Magazine - Q2 2023 - 19
IEEE Circuits and Systems Magazine - Q2 2023 - 20
IEEE Circuits and Systems Magazine - Q2 2023 - 21
IEEE Circuits and Systems Magazine - Q2 2023 - 22
IEEE Circuits and Systems Magazine - Q2 2023 - 23
IEEE Circuits and Systems Magazine - Q2 2023 - 24
IEEE Circuits and Systems Magazine - Q2 2023 - 25
IEEE Circuits and Systems Magazine - Q2 2023 - 26
IEEE Circuits and Systems Magazine - Q2 2023 - 27
IEEE Circuits and Systems Magazine - Q2 2023 - 28
IEEE Circuits and Systems Magazine - Q2 2023 - 29
IEEE Circuits and Systems Magazine - Q2 2023 - 30
IEEE Circuits and Systems Magazine - Q2 2023 - 31
IEEE Circuits and Systems Magazine - Q2 2023 - 32
IEEE Circuits and Systems Magazine - Q2 2023 - 33
IEEE Circuits and Systems Magazine - Q2 2023 - 34
IEEE Circuits and Systems Magazine - Q2 2023 - 35
IEEE Circuits and Systems Magazine - Q2 2023 - 36
IEEE Circuits and Systems Magazine - Q2 2023 - 37
IEEE Circuits and Systems Magazine - Q2 2023 - 38
IEEE Circuits and Systems Magazine - Q2 2023 - 39
IEEE Circuits and Systems Magazine - Q2 2023 - 40
IEEE Circuits and Systems Magazine - Q2 2023 - 41
IEEE Circuits and Systems Magazine - Q2 2023 - 42
IEEE Circuits and Systems Magazine - Q2 2023 - 43
IEEE Circuits and Systems Magazine - Q2 2023 - 44
IEEE Circuits and Systems Magazine - Q2 2023 - 45
IEEE Circuits and Systems Magazine - Q2 2023 - 46
IEEE Circuits and Systems Magazine - Q2 2023 - 47
IEEE Circuits and Systems Magazine - Q2 2023 - 48
IEEE Circuits and Systems Magazine - Q2 2023 - 49
IEEE Circuits and Systems Magazine - Q2 2023 - 50
IEEE Circuits and Systems Magazine - Q2 2023 - 51
IEEE Circuits and Systems Magazine - Q2 2023 - 52
IEEE Circuits and Systems Magazine - Q2 2023 - 53
IEEE Circuits and Systems Magazine - Q2 2023 - 54
IEEE Circuits and Systems Magazine - Q2 2023 - 55
IEEE Circuits and Systems Magazine - Q2 2023 - 56
IEEE Circuits and Systems Magazine - Q2 2023 - 57
IEEE Circuits and Systems Magazine - Q2 2023 - 58
IEEE Circuits and Systems Magazine - Q2 2023 - 59
IEEE Circuits and Systems Magazine - Q2 2023 - 60
IEEE Circuits and Systems Magazine - Q2 2023 - 61
IEEE Circuits and Systems Magazine - Q2 2023 - 62
IEEE Circuits and Systems Magazine - Q2 2023 - 63
IEEE Circuits and Systems Magazine - Q2 2023 - 64
IEEE Circuits and Systems Magazine - Q2 2023 - 65
IEEE Circuits and Systems Magazine - Q2 2023 - 66
IEEE Circuits and Systems Magazine - Q2 2023 - 67
IEEE Circuits and Systems Magazine - Q2 2023 - 68
IEEE Circuits and Systems Magazine - Q2 2023 - 69
IEEE Circuits and Systems Magazine - Q2 2023 - 70
IEEE Circuits and Systems Magazine - Q2 2023 - 71
IEEE Circuits and Systems Magazine - Q2 2023 - 72
IEEE Circuits and Systems Magazine - Q2 2023 - Cover3
IEEE Circuits and Systems Magazine - Q2 2023 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com