IEEE Solid-State Circuits Magazine - Fall 2017 - 63

networks [29] that perform classifications in several optional stages. At
each stage, only a few layers of the network are executed, after which a classification layer tries to guess the class
from the current outputs. Additional
network layers and classifiers are run
only if the obtained probabilities are
not outspoken enough, until a classification with distinct probabilities is
obtained. Such dynamic evaluations
can be performed on any hardware
platform but, again, benefit significantly from implementation-aware
training techniques or topologyopt imized implementations. Inference on the ImageNet data set [29]
required up to 2.6 times fewer operations than state-of-the-art networks at
equivalent accuracy.

Enhancing and Exploiting Sparsity
Deep neural networks exhibit extreme
sparsity, i.e., many of the weight values, as well as intermediate data values, are zero. Figure 11(a) shows the
sparsity of an AlexNet in function
of the used fixed-point word length
within the network. As can be seen,
even for large word lengths, more
than 70% of the activations are zero.
At reduced bit-width computations,
also many weight values are quantized to zero. This opens up many
opportunities.
On the hardware side, this can be
exploited by preventing any MAC with
a zero-valued input [see Figure 11(b)],
by not even fetching zero-valued data
values from memory, and by strongly

An important difference with general-purpose
solutions, however, is that the sizes of the
memories in the hierarchy can be optimized
toward the network's structure.
compressing the on/off chip data
stream using, e.g., Huffman or other
types of encoding. Several hardware
implementations exploit these CNN
characteristics. The authors of [24]
and [11] skip all unnecessary sparse
operations by gating the inputs to
their arithmetic units if the input data
is zero, as a multiply-accumulate with
zero does not change the internal
accumulation result. Both implementations also compress off-chip data
streams, either through run-length
encoding [14] or through a simplified Huffman scheme [23]. The architectures presented in [30] and [31],
on the other hand, allow speeding up
sparse network evaluation s by only
scheduling non-zero operations for
execution, improving computational
throughput up to 1.52 and 5.2 times,
respectively.
More powerful opportunities arise,
again, when the hardware and algorithmic plane are jointly involved.
Deep network training algorithms
can be modified to enhance the network's sparsity by iteratively pruning
the smallest weight values (quantizing them to zero) and retraining the
network [32]. Going one step further,
energy-aware pruning techniques
even take the energy consumption

0
00
0
00
0

Mean Sparsity (%)

100

AlexNet

MAC Array
× × × ×
+ + + +

Weight
Memory
DRAM

model of the hardware into account
and start pruning the layers that consume the most energy, to maximize
pruning efficacy [33]. This easily
allows the pruning of 70-90% of the
weights and saves up to 70% of energy consumption.
Interestingly enough, networks
have more compression capabilities beyond simply that of pruning
low-valued weights. After pruning
and quantizing a network, it turns
out that the resulting weight values
are highly clustered. This allows, e.g.,
the clustering of 8-b weights in only
16 (24) different weight clusters, each
of which can share a common weight
value expressed by a 4-b label. For
every weight value, only the 4-b labels
are stored, and these are expanded
online to their original 8-b value using
a small embedded lookup table.
Recent work has shown that the
combination of pruning, weight sharing, and Huffman compression compresses state-of-the-art networks by
50 times in memory size (deep compression [32]). Traditional accelerators
can benefit from such compression
but only in terms of a reduction
in memory size and the amount of
memory accessed. To execute convolutional operations, they must still

Input/Output
Memory

Layer Inputs
Weights
2
4
6
8
Fixed Point Precision (bits)
(a)

×
+
×
+

Compress
Off-Chip
Communication

Prevent Fetching
Zero-Valued
Data
(b)

×
+
×
+

× ×
+ +
× ×
+ +

Prevent Executing
Zero-Input
MACs

FIGURE 11: (a) The sparsity of input and weight values of a typical network in function of computational precision at which the network is
evaluated. (b) This sparsity allows energy to be saved in the processor's input/output interface, on-chip memories, and data path.

IEEE SOLID-STATE CIRCUITS MAGAZINE

FA L L 2 0 17

Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Fall 2017