IEEE Circuits and Systems Magazine - Q2 2021 - 79

accommodate for the pruned weights. One downside
of these techniques is the increased training complexity,
requiring additional training iterations and the calculation
of second-order information.
Another pruning approach involves penalizing nonzero
parameters in the training stage with additional
regularization terms [52]. After training, small weights
can again be pruned and the network retrained. This approach
can reduce the iterations needed during retraining,
reducing the overall training cost.
Although the usual goal is to preserve a network's accuracy
during pruning, more aggressive pruning strategies
can be considered that balance computational cost
and network accuracy. In Slimmable Neural Networks
[111], authors introduced a method to train multiple versions
of a network with different channel depth. Then,
at runtime, such networks can adjust their width on the
fly based on the on-device resource constraints and preferred
accuracy.
Several sparse network encoding schemes have been
developed that are tuned to particular hardware platform
characteristics. For instance, bitmap-based block
shape storage format can be more suitable than a Coordinate
format (COO) [112] when used to encode sparse
matrices on GPU Tensor Core Units (TCU) [8], [9] (Figure
2). Efficient storage formats can also improve memory
bandwidth. Tensaurus [100] co-designs DL hardware
with a sparse storage format, enabling vectorized
and streaming access of weight data and maximizing
the utilization of the memory bandwidth. In [40], the authors
argue that while the weight matrix is sparse, the
activation matrix is still dense. Therefore, by storing
layer outputs in CHW format instead of HWC (H: height,
W: width, C: channels), vector loads from the activation
matrix make it possible to process multiple spatial
locations simultaneously. Their work showed that this
technique could reduce the number of floating point operations
required by MobileNet [61] and EfficientNet
Matrix in Dense Representation
16-Bit Bitmap Tile
Index
1
a1
a2
a6
a3
a4
a5
a7
1
a8
a9
Row Indices
Column Indices
Index
Bitmap (hex)
COO Storage Format
Row
Column
Data
1
a1
1
2
a2
1
4
a3
1
5
a4
2
5
a5
3
1
a6
3
7
a7
6
4
a8
6
7
a9
Figure 2. COO vs. Bitmap [112]. While COO has no concept of rectangular structures, Bitmap is a more efficient storage format
for the structured sparsity used in [112]. A 8 × 8 matrix in dense (top left) and bitmap formats (green table). Tiles of 4 × 4 partition
the 8 × 8 matrix in a 2 × 2 grid of tiles. Non-zero elements a3, a4, a5, a7 of the circled tile are represented as " 1 " in the bitmap.
The NZ elements of the tile in consecutive locations in the element array. Index points to the first element of the tile. At the bottom,
COO storage format of the same matrix is represented.
SECOND QUARTER 2021
IEEE CIRCUITS AND SYSTEMS MAGAZINE
79
11
1
1
Bitmap Storage Format
0x2042
1
3
0x8230
1
1
7
0x0900
. . .. . .
a3 a4 a5 a7
Element Array

IEEE Circuits and Systems Magazine - Q2 2021

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021