IEEE Computational Intelligence Magazine - August 2021 - 16

B. Multi-Objective Optimization and Speedup
Instead of pruning the model directly in one step, a more effective
approach employed is to prune the model in multiple steps.
If pruning the model in one step, the accuracy will decrease
apparently, and it will be too difficult to restore the model [44].
Figure 3 demonstrates the comparison between the multi-step
pruning method and the single-step pruning method. The
model compression of a well-trained VGG-16 model is tested
on the CIFAR-10 dataset [15], [42]. For the multi-step pruning,
it gradually increases the pruning amount from 0 to 95%
in 32 steps. In each step, the model is pruned partially and retrained
by one epoch. In the single-step pruning, the model is
pruned by 95% immediately, and then re-trained by 32 epochs.
As shown in Figure 3, it can be seen that the multi-step pruning
method outperforms the single-step pruning method in
terms of accuracy with a large margin.
0.25
0.5
0.75
1
04 8121620242832
Time (Epoch)
Multi-Step Pruning
Single-Step Pruning
FIGURE 3 The comparison between multi-step pruning and single-step
pruning, tested on CIFAR-10 using VGG-16 (figure adopted from [15]).
0.25
0.5
0.75
1
A challenge in the multi-step pruning process is that it usually
has high computational complexity. Specifically, each step
requires fine-tuning the model by one or several epochs. If one
attempts to find the optimal pruning amount and quantization
depth for a model, the multi-step pruning process will considerably
delay the optimization progress. To obtain the accuracy
of the compressed model at a given pruning amount and quantization
depth, the model needs to be compressed first, which
usually includes many training epochs. Due to the large search
space, it is almost impossible to pre-store all the compressed
models under any combinations of pruning amount and quantization
depth. For example, the parameters in each layer of the
model can be quantized from 23 bits to 1 bit. The pruning
amount in each layer can range from 0 to 100%. In general, an
L-layer model can have 100 23L
#
pruning amount and quantization depth, assuming 1% pruning
amount granularity.
The EMO technique is adopted to solve this problem.
However, since an evolutionary algorithm is essentially a stochastic
search, it may need thousands of trials (candidate solutions)
to find a high-quality solution. Once a new solution
(architecture) is produced, it takes a substantial amount of time
to perform the training for the evaluation. Consequently, it
may make the EMO-based search impossible.
To address this issue, by taking advantage of the orthogonalPruning
Amount p
Prune Model
Based on p
Save
Pre-Pruned Models
Library
p = p + ρ
Energy Estimator
Energy
Consumption
(a)
(b)
Pruning Amount p
Pre-Pruned Models
Library
Load
Quantizing Depth q
Quantize Model
Based on q
Model Inference
ity between pruning and quantization [45], a two-stage pruning
and quantization co-optimization method is proposed,
which can effectively reduce the computational cost. Specifically,
the optimization process is divided into two stages. In the
first stage, it prunes the model by multiple independent loops.
In each loop, it starts from a well-trained model, prunes the
model with a different pruning amount, fine-tunes the model,
and saves the pruned model into a library. The set of pruning
amounts cover all the possible pruning
amounts which can be referenced by
the multi-objective solver. This is to
guarantee that no pruning process is
required in the second stage. In the second
stage, the multi-objective solver
starts to explore the design space and
tries to find the optimal combinations
of pruning amount and quantization
depth. During this process, the solver
needs to know the accuracy, energy
consumption, and model size under a
given combination of pruning amount
and quantization depth. At this step, one
just needs to load the corresponding
pruned model from the library and
quantize it.
Figure 4 shows an overview of the
Accuracy
FIGURE 4 The process of the proposed two-stage pruning and quantization co-optimization
method. (a) Stage-I: prune the model and save the pre-pruned models to the library; (b) StageII:
load the pre-pruned model from the library, quantize the parameter, and calculate the accuracy
as well as the energy consumption/the model size.
proposed approach. Instead of pruning
and quantizing the models at the same
time, these two actions are taken into
two different stages. In the first stage, it
16 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
possible combinations of
Non-Zero Weights (%)
Accuracy

IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021