IEEE Computational Intelligence Magazine - May 2023 - 79

been applied in many fields [10], [11],
[12]. However, the search time and
computational resource costs of NAS
algorithms are usually high. Most existing
NAS algorithms rely on validation datasets
to optimize network architectures,
requiring considerable time and intensive
computational resources; for example,
NASNet [13] uses 500 GPUs across four
days.
The architecture search problem is
usually framed as a single-objective optimization
problem that cannot simultaneously
consider multiple objectives [9],
[13], [14]. Most real-world deployments
need high classification performance and
low computational resource usage (e.g.,
model size and computational complexity).
Several manually-designed network
architectures, such as MobileNet [15] and
MobileNetV2 [16],havebeen designedto
reduce computational costs while attaining
high classification performance. Some
recent NAS algorithms based on multiobjective
optimization have been proposed
to develop network architectures that are
easy to calculate and deploy. In [17],
NSGA-Net considered the trade-off
between test accuracy and computational
complexity. In [18], the classification performance
and the number of parameters
were combined as objectives. However,
[17], [18] still require numerous
computational resources and long search
time.In thispaper,alow-cost NAS
(LoNAS) method based on a three-stage
evolutionary algorithm (EA) [19] is proposed.
This approach significantly reduces
search time and computational resource
consumption. Moreover, the classification
performance and the number of parameters
can be effectively balanced in LoNAS.
The main contributions of the proposed
algorithm are summarized as follows.
1) A novel network block called the Reg
Block is proposed in LoNAS. The Reg
Block includes the group convolution
and SENet module, reducing the
number of parameters and improving
the network's classification performance.
A variable-architecture encodingstrategyisdesignedbased
on the
Reg Block to encode the network
architecture. By simultaneously considering
variable groups, group widths,
SENet modules, network lengths, and
pooling layer strides, an expanded
search space can be constructed. Then,
more network architectures with good
performance can be found in the
expanded search space.
2) A training-free proxy based on the
neural tangent kernel (NTK) is
designed to evaluate the performance
of individuals in the population. The
NTK can effectively characterize the
trainability of a network architecture
since the condition number of the
NTK (KN)isnegativelycorrelated
with the test accuracy. As KN can be
computedwithout training, the search
time and computational resources can
be significantly reduced.
3) A three-stage EA based on a multiple-criteria
environmental selection
strategy is proposed to effectively
balance exploration and exploitation.
The environmental selection
process criteria are based on the KN
and individual lifespans. The lifespan
is a property associated with each
individual that indicates the number
of evolution generations an individual
goes through. In the first and
third stages, KN is used as the criterion
for eliminating individuals. In
the second stage, individuals are
selected according to their lifespans.
Furthermore, a set of Reg Block
mutation operators is designed to
evolve the population through all
three stages.
II. Related Works
Manually designing high-performance
network architectures is highly challenging,
requiring considerable domain
knowledge and consuming substantial
time and resources. Compared with the
traditional manually-designed approach,
NAS algorithms can automatically design
diverse and high-performance architectures
without professional domain
knowledge.
Regarding the development ofNAS
algorithms, most works have focused
only on improving the classification performance
to outperform manuallydesigned
algorithms. For example, the
Genetic CNN [20] achieves a 27.87%
top-1 recognition error rate on the
ILSVRC2012 dataset [21], which is better
than the error rates of most manually-designed
network architectures [1],
[3], [22]. However, the Genetic CNN
also has the most parameters, reaching
156M parameters. Large-scale Evolution
[14] achieves good classification
accuracy on the CIFAR-100 dataset but
requires 40.4M parameters, which is
more than those of other network
architectures. In addition, while ASNAS
[23] outperforms most NAS
algorithms in terms of classification performance,
AS-NAS requires a substantial
number of parameters. The network
architectures obtained by the above
algorithms achieve good performance in
classification tasks. However, these networks
typically involve a large number
of parameters, which makes these network
architectures difficult to calculate
and deploy.
In recent years, some NAS algorithms
have focused on multiobjective
NAS to construct effective network
architectures for real-world applications
requiring small size and high accuracy.
Multiobjective NAS algorithms aim to
optimize their accuracy and consider
their resource consumption levels. One
kind of multiobjective NAS algorithm,
such as ProxylessNAS [24] and FBNet
[25], rely on a scalarized objective.
This kind of algorithm simultaneously
encourages good classification performance
while penalizing other objectives.
Gradient-based approaches construct a
regularization loss to control the tradeoff
between accuracy and latency. The
algorithms in this category need to
define preference weight values according
to the priority levels of different
objectives before searching; this requires
a considerable number oftrials, consuming
substantial time and computational
resources. Another category of algorithms
includes heuristic algorithms
based on multiobjective optimization,
such as NSGA-Net [17], MSuNAS [26],
and LEMONADE [18]. The algorithms
in this category seek high-performance
network architectures by simultaneously
moving objectives to approximate the
Pareto frontier. However, they still
MAY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 79
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
