IEEE Circuits and Systems Magazine - Q2 2021 - 79
accommodate for the pruned weights. One downside
of these techniques is the increased training complexity,
requiring additional training iterations and the calculation
of second-order information.
Another pruning approach involves penalizing nonzero
parameters in the training stage with additional
regularization terms [52]. After training, small weights
can again be pruned and the network retrained. This approach
can reduce the iterations needed during retraining,
reducing the overall training cost.
Although the usual goal is to preserve a network's accuracy
during pruning, more aggressive pruning strategies
can be considered that balance computational cost
and network accuracy. In Slimmable Neural Networks
[111], authors introduced a method to train multiple versions
of a network with different channel depth. Then,
at runtime, such networks can adjust their width on the
fly based on the on-device resource constraints and preferred
accuracy.
Several sparse network encoding schemes have been
developed that are tuned to particular hardware platform
characteristics. For instance, bitmap-based block
shape storage format can be more suitable than a Coordinate
format (COO) [112] when used to encode sparse
matrices on GPU Tensor Core Units (TCU) [8], [9] (Figure
2). Efficient storage formats can also improve memory
bandwidth. Tensaurus [100] co-designs DL hardware
with a sparse storage format, enabling vectorized
and streaming access of weight data and maximizing
the utilization of the memory bandwidth. In [40], the authors
argue that while the weight matrix is sparse, the
activation matrix is still dense. Therefore, by storing
layer outputs in CHW format instead of HWC (H: height,
W: width, C: channels), vector loads from the activation
matrix make it possible to process multiple spatial
locations simultaneously. Their work showed that this
technique could reduce the number of floating point operations
required by MobileNet [61] and EfficientNet
Matrix in Dense Representation
16-Bit Bitmap Tile
Index
1
a1
a2
a6
a3
a4
a5
a7
1
a8
a9
Row Indices
Column Indices
Index
Bitmap (hex)
COO Storage Format
Row
Column
Data
1
a1
1
2
a2
1
4
a3
1
5
a4
2
5
a5
3
1
a6
3
7
a7
6
4
a8
6
7
a9
Figure 2. COO vs. Bitmap [112]. While COO has no concept of rectangular structures, Bitmap is a more efficient storage format
for the structured sparsity used in [112]. A 8 × 8 matrix in dense (top left) and bitmap formats (green table). Tiles of 4 × 4 partition
the 8 × 8 matrix in a 2 × 2 grid of tiles. Non-zero elements a3, a4, a5, a7 of the circled tile are represented as " 1 " in the bitmap.
The NZ elements of the tile in consecutive locations in the element array. Index points to the first element of the tile. At the bottom,
COO storage format of the same matrix is represented.
SECOND QUARTER 2021
IEEE CIRCUITS AND SYSTEMS MAGAZINE
79
11
1
1
Bitmap Storage Format
0x2042
1
3
0x8230
1
1
7
0x0900
. . .. . .
a3 a4 a5 a7
Element Array
IEEE Circuits and Systems Magazine - Q2 2021
Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021
Contents
IEEE Circuits and Systems Magazine - Q2 2021 - Cover1
IEEE Circuits and Systems Magazine - Q2 2021 - Cover2
IEEE Circuits and Systems Magazine - Q2 2021 - Contents
IEEE Circuits and Systems Magazine - Q2 2021 - 2
IEEE Circuits and Systems Magazine - Q2 2021 - 3
IEEE Circuits and Systems Magazine - Q2 2021 - 4
IEEE Circuits and Systems Magazine - Q2 2021 - 5
IEEE Circuits and Systems Magazine - Q2 2021 - 6
IEEE Circuits and Systems Magazine - Q2 2021 - 7
IEEE Circuits and Systems Magazine - Q2 2021 - 8
IEEE Circuits and Systems Magazine - Q2 2021 - 9
IEEE Circuits and Systems Magazine - Q2 2021 - 10
IEEE Circuits and Systems Magazine - Q2 2021 - 11
IEEE Circuits and Systems Magazine - Q2 2021 - 12
IEEE Circuits and Systems Magazine - Q2 2021 - 13
IEEE Circuits and Systems Magazine - Q2 2021 - 14
IEEE Circuits and Systems Magazine - Q2 2021 - 15
IEEE Circuits and Systems Magazine - Q2 2021 - 16
IEEE Circuits and Systems Magazine - Q2 2021 - 17
IEEE Circuits and Systems Magazine - Q2 2021 - 18
IEEE Circuits and Systems Magazine - Q2 2021 - 19
IEEE Circuits and Systems Magazine - Q2 2021 - 20
IEEE Circuits and Systems Magazine - Q2 2021 - 21
IEEE Circuits and Systems Magazine - Q2 2021 - 22
IEEE Circuits and Systems Magazine - Q2 2021 - 23
IEEE Circuits and Systems Magazine - Q2 2021 - 24
IEEE Circuits and Systems Magazine - Q2 2021 - 25
IEEE Circuits and Systems Magazine - Q2 2021 - 26
IEEE Circuits and Systems Magazine - Q2 2021 - 27
IEEE Circuits and Systems Magazine - Q2 2021 - 28
IEEE Circuits and Systems Magazine - Q2 2021 - 29
IEEE Circuits and Systems Magazine - Q2 2021 - 30
IEEE Circuits and Systems Magazine - Q2 2021 - 31
IEEE Circuits and Systems Magazine - Q2 2021 - 32
IEEE Circuits and Systems Magazine - Q2 2021 - 33
IEEE Circuits and Systems Magazine - Q2 2021 - 34
IEEE Circuits and Systems Magazine - Q2 2021 - 35
IEEE Circuits and Systems Magazine - Q2 2021 - 36
IEEE Circuits and Systems Magazine - Q2 2021 - 37
IEEE Circuits and Systems Magazine - Q2 2021 - 38
IEEE Circuits and Systems Magazine - Q2 2021 - 39
IEEE Circuits and Systems Magazine - Q2 2021 - 40
IEEE Circuits and Systems Magazine - Q2 2021 - 41
IEEE Circuits and Systems Magazine - Q2 2021 - 42
IEEE Circuits and Systems Magazine - Q2 2021 - 43
IEEE Circuits and Systems Magazine - Q2 2021 - 44
IEEE Circuits and Systems Magazine - Q2 2021 - 45
IEEE Circuits and Systems Magazine - Q2 2021 - 46
IEEE Circuits and Systems Magazine - Q2 2021 - 47
IEEE Circuits and Systems Magazine - Q2 2021 - 48
IEEE Circuits and Systems Magazine - Q2 2021 - 49
IEEE Circuits and Systems Magazine - Q2 2021 - 50
IEEE Circuits and Systems Magazine - Q2 2021 - 51
IEEE Circuits and Systems Magazine - Q2 2021 - 52
IEEE Circuits and Systems Magazine - Q2 2021 - 53
IEEE Circuits and Systems Magazine - Q2 2021 - 54
IEEE Circuits and Systems Magazine - Q2 2021 - 55
IEEE Circuits and Systems Magazine - Q2 2021 - 56
IEEE Circuits and Systems Magazine - Q2 2021 - 57
IEEE Circuits and Systems Magazine - Q2 2021 - 58
IEEE Circuits and Systems Magazine - Q2 2021 - 59
IEEE Circuits and Systems Magazine - Q2 2021 - 60
IEEE Circuits and Systems Magazine - Q2 2021 - 61
IEEE Circuits and Systems Magazine - Q2 2021 - 62
IEEE Circuits and Systems Magazine - Q2 2021 - 63
IEEE Circuits and Systems Magazine - Q2 2021 - 64
IEEE Circuits and Systems Magazine - Q2 2021 - 65
IEEE Circuits and Systems Magazine - Q2 2021 - 66
IEEE Circuits and Systems Magazine - Q2 2021 - 67
IEEE Circuits and Systems Magazine - Q2 2021 - 68
IEEE Circuits and Systems Magazine - Q2 2021 - 69
IEEE Circuits and Systems Magazine - Q2 2021 - 70
IEEE Circuits and Systems Magazine - Q2 2021 - 71
IEEE Circuits and Systems Magazine - Q2 2021 - 72
IEEE Circuits and Systems Magazine - Q2 2021 - 73
IEEE Circuits and Systems Magazine - Q2 2021 - 74
IEEE Circuits and Systems Magazine - Q2 2021 - 75
IEEE Circuits and Systems Magazine - Q2 2021 - 76
IEEE Circuits and Systems Magazine - Q2 2021 - 77
IEEE Circuits and Systems Magazine - Q2 2021 - 78
IEEE Circuits and Systems Magazine - Q2 2021 - 79
IEEE Circuits and Systems Magazine - Q2 2021 - 80
IEEE Circuits and Systems Magazine - Q2 2021 - 81
IEEE Circuits and Systems Magazine - Q2 2021 - 82
IEEE Circuits and Systems Magazine - Q2 2021 - 83
IEEE Circuits and Systems Magazine - Q2 2021 - 84
IEEE Circuits and Systems Magazine - Q2 2021 - 85
IEEE Circuits and Systems Magazine - Q2 2021 - 86
IEEE Circuits and Systems Magazine - Q2 2021 - 87
IEEE Circuits and Systems Magazine - Q2 2021 - 88
IEEE Circuits and Systems Magazine - Q2 2021 - 89
IEEE Circuits and Systems Magazine - Q2 2021 - 90
IEEE Circuits and Systems Magazine - Q2 2021 - 91
IEEE Circuits and Systems Magazine - Q2 2021 - 92
IEEE Circuits and Systems Magazine - Q2 2021 - 93
IEEE Circuits and Systems Magazine - Q2 2021 - 94
IEEE Circuits and Systems Magazine - Q2 2021 - 95
IEEE Circuits and Systems Magazine - Q2 2021 - 96
IEEE Circuits and Systems Magazine - Q2 2021 - 97
IEEE Circuits and Systems Magazine - Q2 2021 - 98
IEEE Circuits and Systems Magazine - Q2 2021 - 99
IEEE Circuits and Systems Magazine - Q2 2021 - 100
IEEE Circuits and Systems Magazine - Q2 2021 - 101
IEEE Circuits and Systems Magazine - Q2 2021 - 102
IEEE Circuits and Systems Magazine - Q2 2021 - 103
IEEE Circuits and Systems Magazine - Q2 2021 - 104
IEEE Circuits and Systems Magazine - Q2 2021 - 105
IEEE Circuits and Systems Magazine - Q2 2021 - 106
IEEE Circuits and Systems Magazine - Q2 2021 - 107
IEEE Circuits and Systems Magazine - Q2 2021 - 108
IEEE Circuits and Systems Magazine - Q2 2021 - 109
IEEE Circuits and Systems Magazine - Q2 2021 - 110
IEEE Circuits and Systems Magazine - Q2 2021 - 111
IEEE Circuits and Systems Magazine - Q2 2021 - 112
IEEE Circuits and Systems Magazine - Q2 2021 - 113
IEEE Circuits and Systems Magazine - Q2 2021 - 114
IEEE Circuits and Systems Magazine - Q2 2021 - 115
IEEE Circuits and Systems Magazine - Q2 2021 - 116
IEEE Circuits and Systems Magazine - Q2 2021 - 117
IEEE Circuits and Systems Magazine - Q2 2021 - 118
IEEE Circuits and Systems Magazine - Q2 2021 - 119
IEEE Circuits and Systems Magazine - Q2 2021 - 120
IEEE Circuits and Systems Magazine - Q2 2021 - 121
IEEE Circuits and Systems Magazine - Q2 2021 - 122
IEEE Circuits and Systems Magazine - Q2 2021 - 123
IEEE Circuits and Systems Magazine - Q2 2021 - 124
IEEE Circuits and Systems Magazine - Q2 2021 - 125
IEEE Circuits and Systems Magazine - Q2 2021 - 126
IEEE Circuits and Systems Magazine - Q2 2021 - 127
IEEE Circuits and Systems Magazine - Q2 2021 - 128
IEEE Circuits and Systems Magazine - Q2 2021 - Cover3
IEEE Circuits and Systems Magazine - Q2 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com