IEEE Computational Intelligence Magazine - August 2021 - 25

III. FaUNAE
In this section, we elaborate on how our network efficiently
adapts to new datasets with unsupervised learning. We first
introduce an evolution strategy to effectively evolve the
architecture from an existing architecture to a new one. We
then train our network by exploiting the contrastive learning
to discriminate between positive samples and negative
samples. Our framework is shown in Fig. 1.
A. Search Space
For different datasets, our architecture space is different. For
CIFAR10 [33], we search for two kinds of computation
cells, normal cells and reduction cells, to build the final
architecture. The reduction cells are located at 1/3 and 2/3
of the total depth of the network, and the rest are normal
cells. Following [14], the set of operations include skip
connection (identity), 33# convolutions, 55# convolutions,
33# depth-wise separable convolutions, 55#
depth-wise separable convolutions,33# max pooling,
33#
average pooling, and zero. The search space of a cell
consists of operations on all edges.
On ImageNet, we have experimentally determined that
for unsupervised learning, ResNet [34] is better than cellbased
methods for building an architecture space. We denote
this space as {},
X
i
where i represents a given block. Rather
than repeating the bottleneck (building block in ResNet)
with various operations, however, we allow a set of search
blocks shown in Fig.2 (a) with various operations, including
traditional convolution with kernel size {3, 5}, split-attention
convolution (SAConv) [35] with kernel sizes {3, 5} and
radixes {2, 4}. SAConv is a Split-Attention block [35] which
incorporates feature map split attention within the individual
network blocks. Each block divides the feature map into
several groups and fine-grained subgroups or splits, where
the feature representation of each group is determined by
the weighted combination of its split representations. This
reduces the model size by sharing the 11# convolution to
improve efficiency. To enable a direct trade-off between
depth and block size (indicated by the parameters of the
selected operations), we initiate a deeper over-parameterized
network and allow a block to be skipped by adding the
identity operation to the candidate set of its mixed operation.
So the set of operations i
X in the ith block consists of
M 7= operations. With a limited model size, the network
can either choose to be shallower by skipping more blocks
and using larger blocks or choose to be deeper by keeping
more blocks with smaller size.
The initial structure 0a is first manually designed (e.g.,
ResNet-50, without weight parameters) or searched for
by another NAS (e.g., ProxylessNAS [4]) on different
datasets in a supervised manner2, then remapped to the
search space to accelerate the evolution process and make
use of prior knowledge.
2 The evolution is based on unsupervised learning.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 25
INPUT
Mutation and Train
INPUT
OUTPUT
Conv
3×3
Conv
5×5
SAConv
3×3
OUTPUT
K Times
Search Space
OUTPUT
Conv
3×3
Conv
5×5
SAConv
3×3
SAConv
5×5
SAConv
5×5
INPUT
s0
s1
s2
Teacher
Student
Contrastive
Loss
Reduce
Search Space
w(ok,n) = l(ok,n) -ii 2logN
sak
i
Ωi ←Ωi - {arg max w(ok,n)}
i
k
Search Space
OUTPUT
Conv
3×3
Conv
5×5
SAConv
3×3
SAConv
5×5
Conv
3×3
Conv
5×5
SAConv
3×3
SAConv
5×5
t
INPUT
FIGURE 1 The main framework of the proposed Teacher-Student search strategy. Left: Search space of FaUNAE. Middle: Evolution process of FaUNAE. Right: FaUNAE reduces the search space by
eliminating the operations with the least potential.

IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021