IEEE Computational Intelligence Magazine - August 2021 - 69

which have good performance, and can also search and sample
in a search space where the fitness value may be better.
The studies above do not effectively record the evolution information,
which leads to their inability to guide the entire search process
based on past experience. Most of them tend to focus on the
improvement of the search space and evaluation methods, but the
search strategy also affects an algorithm when finding a competitive
architecture. At the same time, the general search strategy loses the
'experience' in the entire search process, resulting in the algorithm
being unable to obtain more favorable architectures. This paper
focuses on finding a suitable way to guide the whole search process.
For this purpose, this paper proposes a self-adaptive mutation based
on blocks to design CNN architectures. The self-adaptive mechanism
has been widely used in EC, but no one has applied it to the
NAS. In addition, the traditional mutation strategy does not focus
on retaining the evolution information. To choose an appropriate
mutation strategy, the self-adaptive mechanism is added into the
evolutionary process. To prevent the phenomenon of population
degradation and slow convergence, a semi-complete binary competition
selection strategy has also been designed and it can sufficiently
prevent the promising individuals in the population from
being easily abandoned. Moreover, the ENAS algorithm proposed
in this paper is completely automatic, which means that it does not
require the user to have rich expert knowledge about CNN architecture
design.
The rest of the paper is organized as follows: Section II introduces
related work. Section III describes the proposed algorithm in
details. Section IV introduces the designs of the experiment and
Section V analyzes the experimental results. Finally, Section VI provides
our conclusions and directions for future work.
II. Related Work
A. Genetic Algorithm
The GA is an evolutionary algorithm inspired by the natural
selection. Because it has gradient-free characteristics, it can be
used to optimize nonconvex and nondifferentiable problems
[21], [34]. In addition, the GA has the characteristic of being
insensitive to local optima, which enables it to find global optima.
The GA usually uses a series of biologically inspired operators
to solve optimization problems, such as crossover and
mutation. Generally speaking, the main steps of GA are as follows:
1) initialize a population; 2) evaluate fitness; 3) execute
crossover and mutation operators; 4) evaluate individuals; 5)
environmental selection. Among them, steps 3) - 5) are controlled
by a loop, and the stop condition is whether the maximum
generation is reached.
In this paper, each individual represents the architecture of
one CNN, and the fitness is the classification accuracy of each
CNN on the validation dataset. The higher the classification
accuracy, the higher the fitness.
B. Block-Based Design Method
Because a constraint represents human intervention in the
search space, the suitable constraints on the coding space are
FIGURE 2 An example of the ResNet block with three convolutional
layers and one skip connection.
very important. A method with a large number of constraints
can easily obtain a good architecture, but it also will be unable
to find a novel architecture. In addition, the size of the search
space greatly affects the efficiency of search techniques. With
the development of the NAS, more researchers use block-based
techniques to define the search space [35], [36], which combines
various types of layers as the basic unit. These block-based
techniques have good performance and fewer parameters are
required to build the architecture.
ResNet and DenseNet are two kinds of DNNs with better
performance in hand-designed CNNs. Their success
depends on the construction of ResNet blocks and
DenseNet blocks, and the block-based method has been
proven to have better effectiveness than the design method
which is composed of layers
[32], [37], [38]. This design
method can solve the problem of gradient disappearance
[39]. Therefore, the block-based search space can achieve
promising prospects. In this paper, ResNet blocks and
DenseNet blocks are used as the basic units to determine the
search space. For the convenience of representation, ResNet
blocks and DenseNet blocks are respectively called RBs and
DBs. The units composed of RBs or DBs are called RUs or
DUs, respectively. Figure 1 shows the structure of an RU.
Figure 2 shows the internal architecture of an RB. In this
RB, there are three convolutional layers, which are called
conv1, conv2, and conv3. In conv1, 1 × 1 convolution kernels
are used to decrease the number of channels of input
features and the computing complexity. In conv2, the 3 × 3
filters are used to learn the characteristics of the input. In
conv3, the filters with the size of 1 × 1 are used to increase
the number of output channels of the previous convolutional
layer, so that the number of input and output channels
is consistent throughout the entire RB. In addition, a
RU
RB
RB
RB
FIGURE 1 An example of the ResNet Unit.
RB
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 69
Input
Conv1
Conv2
Conv3
Output
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
