IEEE Computational Intelligence Magazine - August 2021 - 70

The existing NAS algorithms can be categorized
into three types: 1) reinforcement learning based
methods; 2) gradient based methods; 3) evolutionary
computation based (ENAS) methods.
skip-connection is used to connect the input layer and the
output layer, and when the sizes of input and output channels
are different, some 1 × 1 filters are added to adjust the
size of channels so that the size of the input channels is the
same as the size of output channels. Figure 3 shows an example
of a DB. As shown in the figure, three convolutional layers
are used to form a DB (the number of convolutional layers
in each DB can be set by the users). In this DB, the input of
each convolutional layer contains the output of each previous
layer. In addition, to better control the input and output
size of each convolutional layer, a parameter k is used. For
example, if the input size of the first layer is a, then the output
size of the first layer is a+k, the input size of the second
layer is a+k, and the output size of the second layer is a+2k.
Through similar operations, some DBs can be constructed.
The two hierarchical representation block-based methods
above can not only reduce the search space, but also enable
deeper convolutional layers to learn features in the shallow
layers. This hierarchical representation of input can also
improve the classification accuracy and make the search process
more flexible [37].
C. Self-Adaptive Mechanism
In recent years, self-adaptive mechanisms have
been increasingly used in the evolutionary
computation community. Moreover, many
EC algorithms based on self-adaptive mechanisms
have been proposed [34]. In the early
years of such research, Qin et al. [40] introduced
the self-adaptive mechanism into the
differential evolution (DE) algorithm, combining different
types of DE, and proposed a new algorithm called self-adaptive
DE. The experimental results showed that self-adaptive
DE can obtain higher quality solutions. In recent years, Xue
et al. [24] applied a self-adaptive PSO to solve large-scale feature
selection problems, and experimental results showed its
effectiveness in reducing redundant features. In the field of
neural networks, adaptive mechanisms, such as adaptive gradient
algorithm [41], adadelta [42], and adam [43] have been
applied for selecting the learning rate. These algorithms
can adjust the learning rate to meet the needs of different
learning rates in different training stages. To the best of our
knowledge, there is no work that introduces the adaptive
mechanism into the CNN architecture design. Therefore, this
paper proposes a self-adaptive mutation method to search the
CNNs' architectures so that the algorithm can adaptively
choose appropriate mutation operations, such as adding,
removing, and modifying units.
DB
III. Details of SaMuNet
In this section, the framework of the algorithm is described,
and its main components are discussed in details. A self-adaptive
mutation network (SaMuNet) is proposed in this paper. The
proposed NAS method is mainly used in image classification
tasks. For clarity, the frequently used notations and their
descriptions are shown in Table I.
FIGURE 3 An example of the DenseNet block with three convolutional
layers.
TABLE I Notations and Corresponding Descriptions.
NOTATIONS DESCRIPTIONS
Population size
N
nlter
m
n
Dtrain
Dvalid
p1, p2
q1, q2
Maximal iteration
Crossover probability
Mutation probability
Train datasets
Validation datasets
Two parent individuals
Two offspring individuals
Q
nsflag
nfflag
S
F
COGS
NS
NOTATIONS DESCRIPTIONS
P
Current population
Offspring population
Success matrix in a generation
Fail matrix in a generation
Success matrix in defined generation
Fail matrix in defined generation
Candidate offspring generate strategy
Number of strategies
A. Framework of SaMuNet
The framework of SaMuNet is shown in Algorithm 1, first, a
population with size N is initialized. Then, each individual in the
initialized population is evaluated. After that, the architecture of
the CNN is evolved by the crossover and self-adaptive mutation
operators. A new population will be generated by the environment
selection operator. This
process continues until the
maximum generation number
is reached. Finally, the best
generated individual is decoded
into a CNN with the best
architecture. In the following
parts, encoding strategy, population
initialization, individual
evaluation, crossover operators,
self-adaptive mutation
mechanism, and the proposed
environment selection operator
are introduced.
70 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
Input
Conv1
Conv2
Conv3
Output
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
