IEEE Circuits and Systems Magazine - Q2 2021 - 82

intra-layer connections). Therefore, manually searching
for a good topology may not be a feasible task for many
applications and designers, given the large combinatorial
search space and the long training times associated
with the evaluation of each design point.
Current Network Architecture Search (NAS) algorithms
aim to resolve this problem by providing computationally
feasible search algorithms. However, this
search space can easily be expanded by including hardware-related
parameters such as quantization, weight
sparsification, and platform-specific parameters such
as memory bandwidth. Recently, hardware-aware NAS
algorithms try to include implementation factors as another
constraint for efficiency in their search. For example,
MnasNet [103] considers the model latency for
producing mobile-friendly architectures. Yet, there is
still a lot of room for improvement as currently, a single
(or a limited number of) hardware platform(s) can be included
in NAS [30], [99], [113].
b) Knowledge Distillation
In many cases, best performing supervised learning models
have many parameters and are computationally expensive.
However, smaller networks are more preferred for hardware
realization. Knowledge Distillation is a training method
for model compression that attempts to resolve this issue
by training models with fewer parameters and competitive
inference accuracy compared to the larger ones.
In this method, a smaller network is trained to learn a
larger network's behavior by replicating its output at every
level [59], [109]. The larger network is trainable with
current training methods and has more parameters, hidden
layers, and architectural complexity. Although this
method has its limitations, it is a promising technique to
train smaller networks suitable for edge-computing. For
example, in a modified knowledge distillation method,
a teaching assistant network is used as an intermediate
network to learn from a larger network and teach an even
smaller network [83].
Channels
Channels
Input
Group
Conv. 1
Features
Group
Conv. 2
Output
(a)
(b)
(c)
Channel
Shuffle
Channels
Identity
Block Hadamard
Input
1 × 1 Group Conv.
Channel Shuffle
1 × 1 Group Conv.
Sum
3 × 3 Depth-Wise
Conv.
Input
1 × 1 Group Conv.
Block Hadamard
1 × 1 Group Conv.
Sum
3 × 3 Depth-Wise
Conv.
(d)
(e)
Figure 5. ShuffleNet and its variant, HadaNet. (a) two stacked convolution layers with the same number of groups. Each output
channel only relates to the input channels within the group. No cross talk; (b) input and output channels are fully related when
Group Conv.2 takes data from different groups after Group Conv.1; (c) an equivalent implementation to (b) using channel shuffle;
(d) a ShuffleNet block; (e) HadaNet variation.
82
IEEE CIRCUITS AND SYSTEMS MAGAZINE
SECOND QUARTER 2021

IEEE Circuits and Systems Magazine - Q2 2021

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021

Contents
IEEE Circuits and Systems Magazine - Q2 2021 - Cover1
IEEE Circuits and Systems Magazine - Q2 2021 - Cover2
IEEE Circuits and Systems Magazine - Q2 2021 - Contents
IEEE Circuits and Systems Magazine - Q2 2021 - 2
IEEE Circuits and Systems Magazine - Q2 2021 - 3
IEEE Circuits and Systems Magazine - Q2 2021 - 4
IEEE Circuits and Systems Magazine - Q2 2021 - 5
IEEE Circuits and Systems Magazine - Q2 2021 - 6
IEEE Circuits and Systems Magazine - Q2 2021 - 7
IEEE Circuits and Systems Magazine - Q2 2021 - 8
IEEE Circuits and Systems Magazine - Q2 2021 - 9
IEEE Circuits and Systems Magazine - Q2 2021 - 10
IEEE Circuits and Systems Magazine - Q2 2021 - 11
IEEE Circuits and Systems Magazine - Q2 2021 - 12
IEEE Circuits and Systems Magazine - Q2 2021 - 13
IEEE Circuits and Systems Magazine - Q2 2021 - 14
IEEE Circuits and Systems Magazine - Q2 2021 - 15
IEEE Circuits and Systems Magazine - Q2 2021 - 16
IEEE Circuits and Systems Magazine - Q2 2021 - 17
IEEE Circuits and Systems Magazine - Q2 2021 - 18
IEEE Circuits and Systems Magazine - Q2 2021 - 19
IEEE Circuits and Systems Magazine - Q2 2021 - 20
IEEE Circuits and Systems Magazine - Q2 2021 - 21
IEEE Circuits and Systems Magazine - Q2 2021 - 22
IEEE Circuits and Systems Magazine - Q2 2021 - 23
IEEE Circuits and Systems Magazine - Q2 2021 - 24
IEEE Circuits and Systems Magazine - Q2 2021 - 25
IEEE Circuits and Systems Magazine - Q2 2021 - 26
IEEE Circuits and Systems Magazine - Q2 2021 - 27
IEEE Circuits and Systems Magazine - Q2 2021 - 28
IEEE Circuits and Systems Magazine - Q2 2021 - 29
IEEE Circuits and Systems Magazine - Q2 2021 - 30
IEEE Circuits and Systems Magazine - Q2 2021 - 31
IEEE Circuits and Systems Magazine - Q2 2021 - 32
IEEE Circuits and Systems Magazine - Q2 2021 - 33
IEEE Circuits and Systems Magazine - Q2 2021 - 34
IEEE Circuits and Systems Magazine - Q2 2021 - 35
IEEE Circuits and Systems Magazine - Q2 2021 - 36
IEEE Circuits and Systems Magazine - Q2 2021 - 37
IEEE Circuits and Systems Magazine - Q2 2021 - 38
IEEE Circuits and Systems Magazine - Q2 2021 - 39
IEEE Circuits and Systems Magazine - Q2 2021 - 40
IEEE Circuits and Systems Magazine - Q2 2021 - 41
IEEE Circuits and Systems Magazine - Q2 2021 - 42
IEEE Circuits and Systems Magazine - Q2 2021 - 43
IEEE Circuits and Systems Magazine - Q2 2021 - 44
IEEE Circuits and Systems Magazine - Q2 2021 - 45
IEEE Circuits and Systems Magazine - Q2 2021 - 46
IEEE Circuits and Systems Magazine - Q2 2021 - 47
IEEE Circuits and Systems Magazine - Q2 2021 - 48
IEEE Circuits and Systems Magazine - Q2 2021 - 49
IEEE Circuits and Systems Magazine - Q2 2021 - 50
IEEE Circuits and Systems Magazine - Q2 2021 - 51
IEEE Circuits and Systems Magazine - Q2 2021 - 52
IEEE Circuits and Systems Magazine - Q2 2021 - 53
IEEE Circuits and Systems Magazine - Q2 2021 - 54
IEEE Circuits and Systems Magazine - Q2 2021 - 55
IEEE Circuits and Systems Magazine - Q2 2021 - 56
IEEE Circuits and Systems Magazine - Q2 2021 - 57
IEEE Circuits and Systems Magazine - Q2 2021 - 58
IEEE Circuits and Systems Magazine - Q2 2021 - 59
IEEE Circuits and Systems Magazine - Q2 2021 - 60
IEEE Circuits and Systems Magazine - Q2 2021 - 61
IEEE Circuits and Systems Magazine - Q2 2021 - 62
IEEE Circuits and Systems Magazine - Q2 2021 - 63
IEEE Circuits and Systems Magazine - Q2 2021 - 64
IEEE Circuits and Systems Magazine - Q2 2021 - 65
IEEE Circuits and Systems Magazine - Q2 2021 - 66
IEEE Circuits and Systems Magazine - Q2 2021 - 67
IEEE Circuits and Systems Magazine - Q2 2021 - 68
IEEE Circuits and Systems Magazine - Q2 2021 - 69
IEEE Circuits and Systems Magazine - Q2 2021 - 70
IEEE Circuits and Systems Magazine - Q2 2021 - 71
IEEE Circuits and Systems Magazine - Q2 2021 - 72
IEEE Circuits and Systems Magazine - Q2 2021 - 73
IEEE Circuits and Systems Magazine - Q2 2021 - 74
IEEE Circuits and Systems Magazine - Q2 2021 - 75
IEEE Circuits and Systems Magazine - Q2 2021 - 76
IEEE Circuits and Systems Magazine - Q2 2021 - 77
IEEE Circuits and Systems Magazine - Q2 2021 - 78
IEEE Circuits and Systems Magazine - Q2 2021 - 79
IEEE Circuits and Systems Magazine - Q2 2021 - 80
IEEE Circuits and Systems Magazine - Q2 2021 - 81
IEEE Circuits and Systems Magazine - Q2 2021 - 82
IEEE Circuits and Systems Magazine - Q2 2021 - 83
IEEE Circuits and Systems Magazine - Q2 2021 - 84
IEEE Circuits and Systems Magazine - Q2 2021 - 85
IEEE Circuits and Systems Magazine - Q2 2021 - 86
IEEE Circuits and Systems Magazine - Q2 2021 - 87
IEEE Circuits and Systems Magazine - Q2 2021 - 88
IEEE Circuits and Systems Magazine - Q2 2021 - 89
IEEE Circuits and Systems Magazine - Q2 2021 - 90
IEEE Circuits and Systems Magazine - Q2 2021 - 91
IEEE Circuits and Systems Magazine - Q2 2021 - 92
IEEE Circuits and Systems Magazine - Q2 2021 - 93
IEEE Circuits and Systems Magazine - Q2 2021 - 94
IEEE Circuits and Systems Magazine - Q2 2021 - 95
IEEE Circuits and Systems Magazine - Q2 2021 - 96
IEEE Circuits and Systems Magazine - Q2 2021 - 97
IEEE Circuits and Systems Magazine - Q2 2021 - 98
IEEE Circuits and Systems Magazine - Q2 2021 - 99
IEEE Circuits and Systems Magazine - Q2 2021 - 100
IEEE Circuits and Systems Magazine - Q2 2021 - 101
IEEE Circuits and Systems Magazine - Q2 2021 - 102
IEEE Circuits and Systems Magazine - Q2 2021 - 103
IEEE Circuits and Systems Magazine - Q2 2021 - 104
IEEE Circuits and Systems Magazine - Q2 2021 - 105
IEEE Circuits and Systems Magazine - Q2 2021 - 106
IEEE Circuits and Systems Magazine - Q2 2021 - 107
IEEE Circuits and Systems Magazine - Q2 2021 - 108
IEEE Circuits and Systems Magazine - Q2 2021 - 109
IEEE Circuits and Systems Magazine - Q2 2021 - 110
IEEE Circuits and Systems Magazine - Q2 2021 - 111
IEEE Circuits and Systems Magazine - Q2 2021 - 112
IEEE Circuits and Systems Magazine - Q2 2021 - 113
IEEE Circuits and Systems Magazine - Q2 2021 - 114
IEEE Circuits and Systems Magazine - Q2 2021 - 115
IEEE Circuits and Systems Magazine - Q2 2021 - 116
IEEE Circuits and Systems Magazine - Q2 2021 - 117
IEEE Circuits and Systems Magazine - Q2 2021 - 118
IEEE Circuits and Systems Magazine - Q2 2021 - 119
IEEE Circuits and Systems Magazine - Q2 2021 - 120
IEEE Circuits and Systems Magazine - Q2 2021 - 121
IEEE Circuits and Systems Magazine - Q2 2021 - 122
IEEE Circuits and Systems Magazine - Q2 2021 - 123
IEEE Circuits and Systems Magazine - Q2 2021 - 124
IEEE Circuits and Systems Magazine - Q2 2021 - 125
IEEE Circuits and Systems Magazine - Q2 2021 - 126
IEEE Circuits and Systems Magazine - Q2 2021 - 127
IEEE Circuits and Systems Magazine - Q2 2021 - 128
IEEE Circuits and Systems Magazine - Q2 2021 - Cover3
IEEE Circuits and Systems Magazine - Q2 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2023Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2022Q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021Q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2021q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2020q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2019q1
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q4
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q3
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q2
https://www.nxtbook.com/nxtbooks/ieee/circuitsandsystems_2018q1
https://www.nxtbookmedia.com