IEEE Computational Intelligence Magazine - May 2023 - 63

removed without affecting the overall
performance ofthe network [32].Moreover,
some other researchers evaluated
the importance (or saliency) of a neuron
via its impact on the objective function.
For instance, in Optimal Brain Damage
[23] and Optimal Brain Surgeon [33],
the second derivative (Hessian matrix) of
the objective function with respect to the
parameters is used to determine the
redundant weights. Similarly, Lee et al.
introduced a saliency criterion based on
the normalized magnitude ofthe derivatives
in the objective function with
respect to each weight [34].
With advanced pruning techniques,
FIGURE 4. The computing requirements increased by 300,000 times from AlexNet [20] to
AlphaGo Zero [21].
during training to increase weight sparsity
[28]. However, these element-wise
magnitude-based pruning methods may
result in unstructured network organizations
which are difficult to be compressed
or accelerated without specialized hardware
support [29].
Structured pruning approaches, on
the other hand, alleviate the above issue
by pruning the network on the filter or
layer level. For instance, Wen et al. proposed
a Structured Sparsity Learning
(SSL) approach to regularize network
structures (i.e., filters, channels, and
layers). Group Lasso is then used to zero
out the grouped weights [30]. In addition
to the weights, some other indicators
have also been explored for pruning. Liu
et al. considered the scaling factor from
batch normalization layer and pruned
those channels ofsmall scaling factor values
[31]. Hu et al. argued that certain activation
functions such as Rectified Linear
Unit (ReLU) could generate numerous
zeros, and these zero activation neurons
are redundant. Hence, they can be
it is able to significantly reduce network
size, which leads to reduced memory
requirements and less computational
cost in inference during model deployment
stage [35]. However, some pruning
techniques may result in more
training efforts during model building
stage, such as methods that are based on
the procedures oftraining, pruning, and
fine-tuning/re-training [36], [37].A
more efficient way is to perform pruning
at initialization [38] or using sparse
training, i.e., training under sparsity
constraints [39].
FIGURE 5. The process of weight/neuron pruning. The first step is to determine which
connections to be pruned. Taking the magnitude-based pruning [26] as an example, the
connection whose weight amplitude jwj is lower than a preset threshold will be pruned (red
dotted arrow). If all the input or output connections of a neuron are pruned, then the neuron can
also be removed entirely (blue dotted circle). When a connection is pruned, the corresponding
value in the weight matrix will be set to zero.
2)Quantization
Another approach for computation-efficientAIisvia
quantization,where 32-bit
floating-point parameters are quantized
into lower numerical precision of 16bit
[40],8-bit [41] or even 1-bit [42], [43].
Using low-bit numerical representations
not only reduces model storage cost but
also results in significant speed-up during
inference, as the most time-consuming
process, floating point Multiply-Accumulate
(MAC) operations, could be avoided.
Jacob et al. quantized both weights and
activations into 8-bit integers for integerarithmetic-only
hardware [44].Incremental
Network Quantization (INQ) proposed
in [45] converted full-precision
weights into power-of-two values with
lower precision. It allows the replacement
of multiplication operations with bit-shift
operations which are more efficient for
hardware like Field-Programmable Gate
Array (FPGA).
Although quantization is generally
applied to weights and activations, it can
MAY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 63
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
