IEEE Circuits and Systems Magazine - Q2 2021 - 77

We believe that with appropriate infrastructure,
retargetable compilers will be able to target
heterogeneous hardware as well.
accuracy of the trained model. While the cost of training
can be significant, the most limiting resource during
training is usually the expertise of an DL-trained
engineer to ensure that the training process makes
progress. In order to minimize risks associated with
training, it is common to use automated techniques
like Network Architecture Search (NAS) [41] and scale
compute performance by distributing computation
across computer networks.
For deployment, however, a network configuration
is often frozen and will be executed repeatedly. At this
point, additional optimization can reduce the computational
requirements of a network. Table I shows the
memory size and compute requirements of some modern
networks. Storing and executing these networks
can become challenging, especially when multiple networks
need to run concurrently. Embedded systems
often have severe limitations on power consumption
due to battery size and heat dissipation limitations, so
using more memory or a faster processor may simply
not be possible. Larger models also typically require
more memory bandwidth, which can significantly limit
achievable performance.
In the subsections that follow, we review many existing
techniques for optimizing DL networks for efficient
execution. We divide these into three main categories,
focusing on leveraging quantization, sparsity, and codesign
between models and architectures.
A. Quantization
One approach to model optimization is the Quantization
of the weights and activations to use less expensive
floating-point or reduced-precision fixed-point operations
[55], [120]. During training, it is common to use 32-bit
single-precision floating-point rather than 64-bit double-precision,
since most networks can still be trained
effectively at single precision [50]. In some cases, training
can still be performed using 16-bit half-precision or
'bfloat' formats, although in others, convergence on an
optimum may be slowed, and the training process may
not achieve optimal accuracy if the loss-minimizing optimization
process gets 'stuck' in a local minima.
During deployment, more aggressive quantization is
often possible, since deep learning models are fundamentally
designed with generalization and robustness
in mind. As a result, it is often possible to reduce the
precision of computation in a network without signifiSECOND
QUARTER 2021
cantly affecting the results. For instance, many image
processing applications can be trained and then quantized
to 8 bits for inference without much loss in accuracy
[24]. Language processing applications are often
deployed using floating-point, due to higher dynamicrange
requirements. Quantization to very small bitwidths
(i.e. 1 or 2 bits) [93], is also possible for some
networks, but often requires a quantization-aware
training process.
To incorporate quantization during training, a common
technique is to add quantization operations in
the forward path, but to compute the backward path
using floating point. This allows the training process
to continue without getting stuck in a local minimum.
Specialized training frameworks, such as Brevitas [5],
QPyTorch [13], and quantized TensorFlow [10], facilitate
representing these quantization operations. Brevitas, in
particular, is focused on low-bitwidth quantization and
is integrated with the FINN framework [7], [28] to enable
implementing heavily quantized networks in programmable
logic.
One challenge in training quantized networks is identifying
appropriate amounts of quantization for the different
layers of a network. For instance, in quantized image
processing it is common to compute the first layer
of a network in a precision that takes advantage of the
sensitivity and dynamic range of a connected sensor.
Then later layers can be quantized more aggressively.
Table I.
Memory and compute requirements of some modern
networks.
Network
AlexNet [107]
VGG-16 [107]
VGG-19 [107]
ResNet-50 [107]
ResNet-101 [107]
ResNet-152 [107]
GoogleNet [107]
InceptionV3 [21]
MobileNet [21]
SqueezeNet [21]
Model
Size (MB)
233
528
548
98
170
230
27
89
38
30
GFLOPS
0.7
15.5
19.6
3.9
7.6
11.3
1.6
6
0.58
0.84
IEEE CIRCUITS AND SYSTEMS MAGAZINE
77
IEEE Circuits and Systems Magazine - Q2 2021

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021
