IEEE Computational Intelligence Magazine - August 2021 - 14

Evolutionary multi-objective optimization has been
widely used to search for the optimal solutions,
in the presence of trade-offs between multiple
conflicting objectives.
learning to provide the model compression policy, which can
accelerate the inference on mobiles considerably.
Recently, edge devices have become increasingly popular
for AI applications. However, considering the large amount of
energy consumed for the model inference, the deployment of
CNN on edge devices becomes challenging. To solve this
problem, some scholars proposed model compression
approaches to reduce the energy consumption directly, using
quantization [13] and/or pruning [12] techniques. In [12], an
energy-aware network pruning approach is proposed to reduce
the overall energy across all layers by
and 16 . # for GoogLeNet [30].
37 . # for AlexNet [29]
From the above, it can be seen that model compression is
essentially a multi-objective optimization problem, with several
objectives to be considered, including accuracy, energy consumption,
model size, etc. Previous studies rarely deal with
multiple objectives at the same time. A common way adopted
in literature is to optimize only one of the objectives while setting
the remaining ones to be hard constraints. In this work, the
evolutionary multi-objective optimization technique is applied
to tackle these objectives simultaneously.
B. Evolutionary Multi-Objective Optimization
In the real-world systems, there exist plenty of problems having
two or more (often conflicting) objectives which one needs to
consider simultaneously. Such problems are called the multiobjective
optimization problems (MOPs). Without loss of generality,
a multi-objective optimization problem (MOP) can be
formulated as the following minimization problem:
T
min Ff ff
gj J
hk K
x ! X
x
s.t.
j
k
xx xx
x
x
= 12
#!
=
!
01 2
01 2
f
f
f
M
X 3 Rn
x xx xn
() (( ), (),, ())
() ,{ ,, ,},
() ,{ ,, ,},
,
(1)
where J denotes the number of inequality constraints, K is the
number of equality constraints,
= 12 f (, ,, )T
is a candidate solution, and :F X " RM
consists of M (conflicting) objective functions.
Let a and b be two feasible solutions for an MOP defined in
Equat ion (1), one can say that a dominates b if
() ()
67 1 (),
uf fvffandab ab where ,{ ,, ,}.
uu vv
#
()
uv ! 12 Mf
A solution is Pareto optimal if it is not dominated by any other
solutions. Due to the conflict of the objectives in MOPs, there
are a set of Pareto optimal solutions, which represent the best
possible trade-offs among different objectives. The optimal solution
set in the decision space is called the Pareto set (PS), and its
mapping in the objective space is called the Pareto front (PF).
14 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
is the decision space,
In the literature, many approaches have
been developed to solve MOPs since the
1950s [31]. Among them, evolutionary algorithms
(EAs) stand out thanks to the nature of
population-based search that aims to approximate
the whole Pareto front in a single execution.
Also, EAs are typically exempt from the
characteristics of the PF than conventional mathematical
programming techniques [31]. They can handle the
MOPs with discontinuous and non-convex PFs well.
Since the seminal work, called Vector Evaluated Genetic
Algorithm (VEGA) [32], was proposed by Schaffer in 1985, a
large number of multi-objective evolutionary algorithms
(MOEAs) have been developed and adopted in various applications.
In MOEAs, the selection strategy of individuals in the
population plays a key role in the evolutionary process. Since
the optimal solutions are those non-dominated to each other
in the whole search space, Pareto dominance naturally becomes
a viable criterion for selecting promising solutions during the
evolutionary process. The Pareto dominance criterion, however,
may fail to provide sufficient selection pressure, making the
algorithm hard to converge. This situation can be usually
encountered when the objective space is enormous, e.g., in
many-objective optimization problems [33]-[35]. To push the
population towards the PF, Goldberg proposed a mechanism
called Pareto ranking [36] for the selection in MOEAs. A niche
method is then used in the Nondominated Sorting Genetic
Algorithm (NSGA) [37] to maintain stable sub-populations.
Later on, in its new version Nondominated Sorting Genetic
Algorithm-II (NSGA-II) [38], a crowding degree comparison
operator is adopted to make the ranking scheme more effective
and efficient. NSGA-II is widely used to solve MOPs, despite
its limitations in handling the MOPs with more than three
objectives [39]. Recently, many MOEAs tend to consider other
selection strategies since they may converge fast towards the PF,
such as indicator-based MOEAs, decomposition-based
MOEAs, and bi-goal criterion MOEAs [33].
Recently, there have been a few attempts to exploit
MOEAs to search for efficient neural architectures. For
instance, Lu et al. proposed a method, called NSGA-Net [40],
which formulates the neural architecture search as a multiobjective
problem and uses the NSGA-II algorithm to solve it.
NSGA-Net considers two objectives: the classification error
and the computation cost (measured by the number of MACs).
It has achieved promising results compared with other neural
architecture search methods, e.g., DARTS [5] and ENAS [41],
on the CIFAR-10 dataset [42].
This work studies how the evolutionary multi-objective
(EMO) method can be used in model compression, given its
multi-objective nature.
IV. Our Proposed Method
In real-world applications, users usually have different preferences
on the prediction model's objectives, including accuracy,
energy efficiency, model size, etc. In this section, the evolutionary
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
