IEEE Circuits and Systems Magazine - Q3 2023 - 43

systolic array adopts the WS dataflow to reuse cached
weights across all inputs. Jouppi et al. [13] is an example
of ML accelerator that adopts the WS dataflow.
Output Stationary (OS) Dataflow: In the OS dataflow,
a PE stores and accumulates a psum data locally. The OS
dataflow can effectively reduce the amount of reading
and writing of psums from and to the memory. An example
OS dataflow mapped on a PE array is illustrated in
Fig. 9(b). In this example, each PE accumulates a psum,
PP P012
There are multiple benefits by exploiting sparsity in
,, , or P3, locally. In each cycle, new weights are
fetched and sent to the PEs, and one input is broadcast
across all PEs. Each PE computes a MAC and updates its
local psum. Upon completion, the output data are written
back from the PEs to memory. Du et al. [22] and Deng
et al. [23] are examples of ML accelerators that adopt
the OS dataflow.
IV. Sparse Architecture
The continued growth of model size and complexity has
motivated research efforts in leveraging data sparsity to
reduce the compute and storage requirements. In this
section, we present an overview of network sparsity and
how to exploit it to make more efficient processing.
A. Sparsity in Neural Networks
The sparsity in a network comes from both the model's
weights (Ws) and input activations (IAs). For the model
weights, network pruning and other sparsification
techniques can be used to zero out a large number of
weights in a model with only a small inference accuracy
drop [24], [25], [26], [27], [28], [29]. For the input activations,
some commonly-used operators like rectifier
linear unit (ReLU) can clamp all negative activations to
zeros, resulting in sparsity in output activations (OA),
which become input activations (IA) of the next layer.
A CONV computation with IA and W sparsity is illustrated
in Fig. 10. With network pruning [24], the
typical W density (nonzero data over all data)
ranges from 40% to 50% and the IA density
ranges between 30% and 55% for well-known
models, e.g., AlexNet, VGG-16, and ResNet-50
[30]. An up to 38% and 4% density for IA and W,
respectively, is achieved by [24] on the FC layers
of VGG-16. The CONV layers can be pruned
down to 19% and 22% density for IA and W,
respectively. Zhang et al. [26] reported a 95%
W sparsity on AlexNet using ADMM. For an IA
and a W with 50% density each, because the
nonzero W and IA are nearly randomly distributed,
the amount of effectual computation,
i.e., computation that does not involve a zero,
is only about 25%.
THIRD QUARTER 2023
designing DNN compute. First, data sparsity can be exploited
to save power. Accelerators e.g., Eyeriss [32] gate
the computation, e.g., by turning off the clock, whenever
a zero in the IA is detected during processing. This technique
can effectively reduce the power consumption
during DNN processing and can be conveniently incorporated
into existing dense DNN accelerators. However,
the throughput remains the same since PEs become idle
during ineffective computation.
Second, data sparsity can be used to reduce off-chip
memory storage and bandwidth usage. The sparse W
and IA can be stored in a compressed format with only
nonzero elements. They are loaded and decompressed
for computation. The compressed storage reduces the
storage size and memory bandwidth. However, the decompression
can be difficult to parallelize and costly in
power and area, leading to a bottleneck and additional
overhead for DNN processing.
Lastly, data sparsity can be used to reduce latency
by skipping the ineffectual computation. During processing,
IA-W pairs are identified by searching through
the sparse IA and W data and sent to the compute. The
search step avoids wasting time on unnecessary computation,
resulting in significant latency savings. Stateof-the-art
sparse DNN accelerators [31], [33], [34], [35],
[36] process data directly in the compressed form, offering
both low memory bandwidth and high degree of acceleration.
However, supporting sparse processing can
cost a high design complexity.
B. Sparse Compression Format
Sparse compression formats are used to store sparse
data in compact ways to save storage space. A compressed
format contains only nonzero data values and
metadata to hold the information for locating the positions
of nonzero values in the uncompressed vectors and
matrices. During processing, the metadata is decoded to
Figure 10. Convolution computation between unstructured sparse IA
and W in a sparse DNN. The colored cells indicate nonzero entries,
and the white cells indicate zero entries. Adopted from [31] ©2021
IEEE.
IEEE CIRCUITS AND SYSTEMS MAGAZINE
43
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
