IEEE Circuits and Systems Magazine - Q2 2021 - 77
We believe that with appropriate infrastructure,
retargetable compilers will be able to target
heterogeneous hardware as well.
accuracy of the trained model. While the cost of training
can be significant, the most limiting resource during
training is usually the expertise of an DL-trained
engineer to ensure that the training process makes
progress. In order to minimize risks associated with
training, it is common to use automated techniques
like Network Architecture Search (NAS) [41] and scale
compute performance by distributing computation
across computer networks.
For deployment, however, a network configuration
is often frozen and will be executed repeatedly. At this
point, additional optimization can reduce the computational
requirements of a network. Table I shows the
memory size and compute requirements of some modern
networks. Storing and executing these networks
can become challenging, especially when multiple networks
need to run concurrently. Embedded systems
often have severe limitations on power consumption
due to battery size and heat dissipation limitations, so
using more memory or a faster processor may simply
not be possible. Larger models also typically require
more memory bandwidth, which can significantly limit
achievable performance.
In the subsections that follow, we review many existing
techniques for optimizing DL networks for efficient
execution. We divide these into three main categories,
focusing on leveraging quantization, sparsity, and codesign
between models and architectures.
A. Quantization
One approach to model optimization is the Quantization
of the weights and activations to use less expensive
floating-point or reduced-precision fixed-point operations
[55], [120]. During training, it is common to use 32-bit
single-precision floating-point rather than 64-bit double-precision,
since most networks can still be trained
effectively at single precision [50]. In some cases, training
can still be performed using 16-bit half-precision or
'bfloat' formats, although in others, convergence on an
optimum may be slowed, and the training process may
not achieve optimal accuracy if the loss-minimizing optimization
process gets 'stuck' in a local minima.
During deployment, more aggressive quantization is
often possible, since deep learning models are fundamentally
designed with generalization and robustness
in mind. As a result, it is often possible to reduce the
precision of computation in a network without signifiSECOND
QUARTER 2021
cantly affecting the results. For instance, many image
processing applications can be trained and then quantized
to 8 bits for inference without much loss in accuracy
[24]. Language processing applications are often
deployed using floating-point, due to higher dynamicrange
requirements. Quantization to very small bitwidths
(i.e. 1 or 2 bits) [93], is also possible for some
networks, but often requires a quantization-aware
training process.
To incorporate quantization during training, a common
technique is to add quantization operations in
the forward path, but to compute the backward path
using floating point. This allows the training process
to continue without getting stuck in a local minimum.
Specialized training frameworks, such as Brevitas [5],
QPyTorch [13], and quantized TensorFlow [10], facilitate
representing these quantization operations. Brevitas, in
particular, is focused on low-bitwidth quantization and
is integrated with the FINN framework [7], [28] to enable
implementing heavily quantized networks in programmable
logic.
One challenge in training quantized networks is identifying
appropriate amounts of quantization for the different
layers of a network. For instance, in quantized image
processing it is common to compute the first layer
of a network in a precision that takes advantage of the
sensitivity and dynamic range of a connected sensor.
Then later layers can be quantized more aggressively.
Table I.
Memory and compute requirements of some modern
networks.
Network
AlexNet [107]
VGG-16 [107]
VGG-19 [107]
ResNet-50 [107]
ResNet-101 [107]
ResNet-152 [107]
GoogleNet [107]
InceptionV3 [21]
MobileNet [21]
SqueezeNet [21]
Model
Size (MB)
233
528
548
98
170
230
27
89
38
30
GFLOPS
0.7
15.5
19.6
3.9
7.6
11.3
1.6
6
0.58
0.84
IEEE CIRCUITS AND SYSTEMS MAGAZINE
77
IEEE Circuits and Systems Magazine - Q2 2021
Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021
Contents
IEEE Circuits and Systems Magazine - Q2 2021 - Cover1
IEEE Circuits and Systems Magazine - Q2 2021 - Cover2
IEEE Circuits and Systems Magazine - Q2 2021 - Contents
IEEE Circuits and Systems Magazine - Q2 2021 - 2
IEEE Circuits and Systems Magazine - Q2 2021 - 3
IEEE Circuits and Systems Magazine - Q2 2021 - 4
IEEE Circuits and Systems Magazine - Q2 2021 - 5
IEEE Circuits and Systems Magazine - Q2 2021 - 6
IEEE Circuits and Systems Magazine - Q2 2021 - 7
IEEE Circuits and Systems Magazine - Q2 2021 - 8
IEEE Circuits and Systems Magazine - Q2 2021 - 9
IEEE Circuits and Systems Magazine - Q2 2021 - 10
IEEE Circuits and Systems Magazine - Q2 2021 - 11
IEEE Circuits and Systems Magazine - Q2 2021 - 12
IEEE Circuits and Systems Magazine - Q2 2021 - 13
IEEE Circuits and Systems Magazine - Q2 2021 - 14
IEEE Circuits and Systems Magazine - Q2 2021 - 15
IEEE Circuits and Systems Magazine - Q2 2021 - 16
IEEE Circuits and Systems Magazine - Q2 2021 - 17
IEEE Circuits and Systems Magazine - Q2 2021 - 18
IEEE Circuits and Systems Magazine - Q2 2021 - 19
IEEE Circuits and Systems Magazine - Q2 2021 - 20
IEEE Circuits and Systems Magazine - Q2 2021 - 21
IEEE Circuits and Systems Magazine - Q2 2021 - 22
IEEE Circuits and Systems Magazine - Q2 2021 - 23
IEEE Circuits and Systems Magazine - Q2 2021 - 24
IEEE Circuits and Systems Magazine - Q2 2021 - 25
IEEE Circuits and Systems Magazine - Q2 2021 - 26
IEEE Circuits and Systems Magazine - Q2 2021 - 27
IEEE Circuits and Systems Magazine - Q2 2021 - 28
IEEE Circuits and Systems Magazine - Q2 2021 - 29
IEEE Circuits and Systems Magazine - Q2 2021 - 30
IEEE Circuits and Systems Magazine - Q2 2021 - 31
IEEE Circuits and Systems Magazine - Q2 2021 - 32
IEEE Circuits and Systems Magazine - Q2 2021 - 33
IEEE Circuits and Systems Magazine - Q2 2021 - 34
IEEE Circuits and Systems Magazine - Q2 2021 - 35
IEEE Circuits and Systems Magazine - Q2 2021 - 36
IEEE Circuits and Systems Magazine - Q2 2021 - 37
IEEE Circuits and Systems Magazine - Q2 2021 - 38
IEEE Circuits and Systems Magazine - Q2 2021 - 39
IEEE Circuits and Systems Magazine - Q2 2021 - 40
IEEE Circuits and Systems Magazine - Q2 2021 - 41
IEEE Circuits and Systems Magazine - Q2 2021 - 42
IEEE Circuits and Systems Magazine - Q2 2021 - 43
IEEE Circuits and Systems Magazine - Q2 2021 - 44
IEEE Circuits and Systems Magazine - Q2 2021 - 45
IEEE Circuits and Systems Magazine - Q2 2021 - 46
IEEE Circuits and Systems Magazine - Q2 2021 - 47
IEEE Circuits and Systems Magazine - Q2 2021 - 48
IEEE Circuits and Systems Magazine - Q2 2021 - 49
IEEE Circuits and Systems Magazine - Q2 2021 - 50
IEEE Circuits and Systems Magazine - Q2 2021 - 51
IEEE Circuits and Systems Magazine - Q2 2021 - 52
IEEE Circuits and Systems Magazine - Q2 2021 - 53
IEEE Circuits and Systems Magazine - Q2 2021 - 54
IEEE Circuits and Systems Magazine - Q2 2021 - 55
IEEE Circuits and Systems Magazine - Q2 2021 - 56
IEEE Circuits and Systems Magazine - Q2 2021 - 57
IEEE Circuits and Systems Magazine - Q2 2021 - 58
IEEE Circuits and Systems Magazine - Q2 2021 - 59
IEEE Circuits and Systems Magazine - Q2 2021 - 60
IEEE Circuits and Systems Magazine - Q2 2021 - 61
IEEE Circuits and Systems Magazine - Q2 2021 - 62
IEEE Circuits and Systems Magazine - Q2 2021 - 63
IEEE Circuits and Systems Magazine - Q2 2021 - 64
IEEE Circuits and Systems Magazine - Q2 2021 - 65
IEEE Circuits and Systems Magazine - Q2 2021 - 66
IEEE Circuits and Systems Magazine - Q2 2021 - 67
IEEE Circuits and Systems Magazine - Q2 2021 - 68
IEEE Circuits and Systems Magazine - Q2 2021 - 69
IEEE Circuits and Systems Magazine - Q2 2021 - 70
IEEE Circuits and Systems Magazine - Q2 2021 - 71
IEEE Circuits and Systems Magazine - Q2 2021 - 72
IEEE Circuits and Systems Magazine - Q2 2021 - 73
IEEE Circuits and Systems Magazine - Q2 2021 - 74
IEEE Circuits and Systems Magazine - Q2 2021 - 75
IEEE Circuits and Systems Magazine - Q2 2021 - 76
IEEE Circuits and Systems Magazine - Q2 2021 - 77
IEEE Circuits and Systems Magazine - Q2 2021 - 78
IEEE Circuits and Systems Magazine - Q2 2021 - 79
IEEE Circuits and Systems Magazine - Q2 2021 - 80
IEEE Circuits and Systems Magazine - Q2 2021 - 81
IEEE Circuits and Systems Magazine - Q2 2021 - 82
IEEE Circuits and Systems Magazine - Q2 2021 - 83
IEEE Circuits and Systems Magazine - Q2 2021 - 84
IEEE Circuits and Systems Magazine - Q2 2021 - 85
IEEE Circuits and Systems Magazine - Q2 2021 - 86
IEEE Circuits and Systems Magazine - Q2 2021 - 87
IEEE Circuits and Systems Magazine - Q2 2021 - 88
IEEE Circuits and Systems Magazine - Q2 2021 - 89
IEEE Circuits and Systems Magazine - Q2 2021 - 90
IEEE Circuits and Systems Magazine - Q2 2021 - 91
IEEE Circuits and Systems Magazine - Q2 2021 - 92
IEEE Circuits and Systems Magazine - Q2 2021 - 93
IEEE Circuits and Systems Magazine - Q2 2021 - 94
IEEE Circuits and Systems Magazine - Q2 2021 - 95
IEEE Circuits and Systems Magazine - Q2 2021 - 96
IEEE Circuits and Systems Magazine - Q2 2021 - 97
IEEE Circuits and Systems Magazine - Q2 2021 - 98
IEEE Circuits and Systems Magazine - Q2 2021 - 99
IEEE Circuits and Systems Magazine - Q2 2021 - 100
IEEE Circuits and Systems Magazine - Q2 2021 - 101
IEEE Circuits and Systems Magazine - Q2 2021 - 102
IEEE Circuits and Systems Magazine - Q2 2021 - 103
IEEE Circuits and Systems Magazine - Q2 2021 - 104
IEEE Circuits and Systems Magazine - Q2 2021 - 105
IEEE Circuits and Systems Magazine - Q2 2021 - 106
IEEE Circuits and Systems Magazine - Q2 2021 - 107
IEEE Circuits and Systems Magazine - Q2 2021 - 108
IEEE Circuits and Systems Magazine - Q2 2021 - 109
IEEE Circuits and Systems Magazine - Q2 2021 - 110
IEEE Circuits and Systems Magazine - Q2 2021 - 111
IEEE Circuits and Systems Magazine - Q2 2021 - 112
IEEE Circuits and Systems Magazine - Q2 2021 - 113
IEEE Circuits and Systems Magazine - Q2 2021 - 114
IEEE Circuits and Systems Magazine - Q2 2021 - 115
IEEE Circuits and Systems Magazine - Q2 2021 - 116
IEEE Circuits and Systems Magazine - Q2 2021 - 117
IEEE Circuits and Systems Magazine - Q2 2021 - 118
IEEE Circuits and Systems Magazine - Q2 2021 - 119
IEEE Circuits and Systems Magazine - Q2 2021 - 120
IEEE Circuits and Systems Magazine - Q2 2021 - 121
IEEE Circuits and Systems Magazine - Q2 2021 - 122
IEEE Circuits and Systems Magazine - Q2 2021 - 123
IEEE Circuits and Systems Magazine - Q2 2021 - 124
IEEE Circuits and Systems Magazine - Q2 2021 - 125
IEEE Circuits and Systems Magazine - Q2 2021 - 126
IEEE Circuits and Systems Magazine - Q2 2021 - 127
IEEE Circuits and Systems Magazine - Q2 2021 - 128
IEEE Circuits and Systems Magazine - Q2 2021 - Cover3
IEEE Circuits and Systems Magazine - Q2 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com