IEEE Circuits and Systems Magazine - Q2 2021 - 78

The Xilinx Versal architecture includes specialized on-chip
network resources for data-distribution and an array
of Vector-VLIW processor cores.
However, quantizing all layers of a network equally does
not necessarily achieve the best result. In addition, it
may be helpful to train a network initially with higher
precision and then to increase quantization only after a
network is mostly trained.
One approach to introducing quantization during training
involves changing bit-precision based on a performance
criterion. In [115], networks are trained with two
different bit-widths. Each layer first applies its filters with
low-precision parameters, and then a gating scheme compares
the results against a learnable threshold and decides
if an update stage has to be computed or skipped. Performing
the update stage is equal to applying full-precision filters
to the input.
Note that in some cases, it is also beneficial to quantize
weights and activations differently. Since in most
networks, weights must be stored off-chip while activations
from a previous layer are stored on-chip, weight
quantization has a much bigger effect on memory bandwidth
than activation quantization [65], [95].
B. Pruning and Static Sparsity
Pruning is another optimization that can reduce the
computational complexity of DL networks. Pruning leverages
the redundancy in the representation of many
networks by replacing small weight values with zero.
Huang et al. [63] showed that more than 90% of the
weights could be removed for some networks without
a drop in accuracy. This reduces the storage capacity
required to implement the network, and also allows the
elimination of 80% of the associated compute operations.
Pruning introduces static sparsity in networks,
since this structure is fixed in the weights as part of the
training of a network. This can enable significant architectural
optimization during the inference process,
since multiplications by zero will always have a zero
output. As with quantization, pruning is often incorporated
into the training process, enabling improved
model accuracy. Pruning techniques are an active area
of research [43] and new approaches are still emerging
[44], [77], [105].
Early works such as Optimal Brain Damage [37]
and Optimal Brain Surgeon [53] focus on iteratively retraining
a network to encourage sparsity. At each iteration,
these techniques compute the Hessian of the loss
function to identity low importance weights, assigning
them to zero. After multiple iterations, the introduced
zeros tend to remain, while other weights adjust to
(a)
(b)
(c)
(d)
Figure 1. Static sparsity: weights in a neural network (a), can be sparsified by pruning. (b) shows a general and unstructured pruning
of the parameters, (c) shows channel pruning, and (d) demonstrates an example of per-channel structured pruning.
78
IEEE CIRCUITS AND SYSTEMS MAGAZINE
SECOND QUARTER 2021

IEEE Circuits and Systems Magazine - Q2 2021

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021