IEEE Computational Intelligence Magazine - August 2021 - 30

in 3.1) on ImageNet. We set the initial structure 0a as a random
structure, ResNet50, and the structure searched by Proexyless
on ImageNet100, respectively, to show the importance
of prior knowledge. During the architecture search, the 128M
training samples of ImageNet are divided into two subsets,
80% for the training set, and the remainder for the validation
set for mutation validation and search space reduction. We set
the channel as half of that of ResNet50 for efficiency and pay
attention to the evolution of the operation rather than the
channel. We set the hyperparameter
the epoch of the search is R =2
m kE .
M
)
and 2m are set to 0.9 and 0.3. After evolution, we train the
searched network on ImageNet in an unsupervised manner
for 200 epochs.
C. Results for Classification
In this section, we perform experiments on CIFAR10 and
ImageNet. In the experiment using CIFAR10, every time we
train an epoch, we immediately verify the performance on the
validation set. In the experiment using ImageNet, we perform
unsupervised pretraining and then train a supervised linear classifier
for 100 epochs. Finally, we report top-1 classification
accuracy on the validation set.
E ,3= so the number of
The hyperparameter 1m
1) Results on CIFAR10
On CIFAR10, we use an experimental setting described in
MoCo v1 [1], not the settings used in v2 [11] for ImageNet. In
addition, we use cell-based search space. In the search process,
we search for normal cells and reduction cells. The total number
of layers is set to 6, and reduction cells are located at 1/3
and 2/3. We set the intermediate nodes to
#
tum to 0.9, the weight decay to 310 4
-
M ,4= the momenand
the initial learnTABLE
1 Comparison under the linear classification protocol on CIFAR10.
ARCHITECTURE METHOD
AMDIMSMALL
AMDIMLARGE
RESNET18
AMDIM [44]
AMDIM [44]
FAUNAE
(CHANNEL = 36) MOCO V1
FAUNAE
(CHANNEL = 100) MOCO V1
ACCURACY (%) PARAMS (M)
89.5
91.2
MOCO V1 [1] 84.09
88.83
92.26
194
626
11.1
9.1
69.8
ing rate to 0.025 described in [14]. We use 90% of the training
set as the training data in the search phase, and 10% of the
training set as the validation data. After the search, we train the
final architecture for 600 epochs on CIFAR10. We set the total
number of cell layers to 10 and the
batch size to 128.
We use two NVIDIA Titan V GPUs
SEARCH
COST (GPU
DAYS)
-
-
-
0.125
0.125
TABLE 2 Comparisons under the linear classification protocol on ImageNet.
ARCHITECTURE METHOD
ACCURACY
(%)
RESNET50
RESNET50
RESNET101
RESNET170WIDER
RESNET50L+AB
AMDIMSMALL
AMDIMLARGE
RESNET50
RESNET50
RESNET50
RANDOM
PROXYLESSNAS
FAUNAE
(RANDOM)
FAUNAE
(RESNET50)
INSTDISC [40]
54.0
LOCALAGG [45] 58.8
CPC V1 [42]
CPC V2 [46]
CMC [47]
AMDIM [44]
AMDIM [44]
MOCO V1 [1]
MOCO V2 [11]
SIMCLR [12]
MOCO V2
MOCO V2
MOCO V2
MOCO V2
FAUNAE
(PROXYLESSNAS) MOCO V2
48.7
65.9
64.1
63.5
68.1
60.6
67.5
66.6
66.2
67.8
67.4
67.8
68.3
PARAMS
(M)
24
24
28
303
47
194
626
24
24
24
23
23
24
24
30
SEARCH COST
(GPU DAYS)
-
-
-
-
-
-
-
-
-
-
-
23.1
15.3
15.3
15.3
SEARCH
METHOD
Manual
Manual
Manual
Evolution
Evolution
to search approximately 1.5 hours. We
set the initial number of channels to 36
and 100 for better performance. It can
clearly be seen in Table 1 whether
it is FaUNAE (channel = 36) or
FaUNAE (channel = 100), the model
size of FaUNAE is smaller than that
of AMDIMlarge
and AMDIM .small
Note that the results in Table 1 are
our best, but they are unstable and
we need additional trials to obtain
re sults. Compared with
AMDIM ,large
SEARCH
METHOD
Manual
Manual
Manual
Manual
Manual
Manual
Manual
Manual
Manual
Manual
Random
Gradient-base
Evolution
Evolution
Evolution
FaUNAE (channel = 100) achieves
not only a better performance (91.2
vs. 92.26), but also has fewer parameters
(626M vs. 69.8M). When compared
with ResNet18, FaUNAE is
slightly better than ResNet in terms
of model size (11.1M vs. 9.1M), and
FaUNAE improved performance by
4.74% (84.09 vs. 88.83).
2) Results on ImageNet
During the search, we set an initial
learning rate of 0.03, a momentum
of 0.9, a weight decay of 0.0001, and
then we use 8 Tesla V100 GPUs to
search approximately 46 hours. In
the process of training a supervised
linear classifier, we set the initial
learning rate as 30 and the weight
decay 0 as described in [1]. Table 2
shows that FaUNAE outperforms
ResNet50, ResNet101, ResNet170,
30 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
