IEEE Computational Intelligence Magazine - August 2021 - 41

regression learning model using all paired training samples within
a reasonable time cost. For example, on NASBench-101, it costs
about 1.7 minutes to train the self-supervised regression learning
model if 380k (the number equals to 90% of the total number of
architectures) of the paired training samples are used. If the model
is trained with all the paired samples
^^ + ##
f .rl
1 423 4230 .5kk
hh
for 300 epochs, it will take about 252 years. Therefore, in our
experiments, a fixed number of paired training samples are randomly
selected to pre-train the model
The number is set to
90% of the total number of the neural architectures in the search
space, i.e., 380k for NASBench-101 and 13.7k for NASBench-201.
The training epoch is 300 and the batch size is 64.
The Adam optimizer [48] is employed to optimize the parameters
of the model
f ,rl
the initial learning rate is 5e 4,weight
decay is 1e 4.and
the
A cosine learning rate schedule [49]
without restart is adopted to anneal down the learning rate to
zero. On NASBench-201, the initial learning rate is 1e 4and
it
is trained for 1000 epochs. Other training details are the same as
NASBench-101.
The self-supervised central contrastive learning utilizes all of
the architectures in NASBench-101 to pre-train the model f .ccl
The training epoch is 300, the regularization weight m is 0.5, and
the temperature x is 0.07. The batch size is 140k, and the training
architectures are 140k. When pre-training on NASBench-201,
the batch size is 10k, and the training architectures
are 1k. For both search spaces, the initial learning rate is 5e 3.Other
training details like the optimizer, weight decay, and the
learning rate schedule are identical with those for self-supervised
regression learning.
The initial learning rates of the supervised neural predictor
on NASBench-101 and NASBench-201 are 5e 3and
1e
3- , respectively. The other training details of the supervised
neural predictor are the same as self-supervised central contrastive
learning.
After pre-training, the neural architectures and their corresponding
validation accuracies are used to fine-tune the neural
predictors. SS-RL and SS-CCL are fine-tuned with an initial
learning rate of 5e 5and
5e 3- , respectively. The weight
decay is 1e 4- . The optimizer and the learning schedule are the
same as the self-supervised pre-training.
3) Setup
The search budget and training epochs of neural predictors
directly affect the time cost of NAS. The supervised neural
predictor, SS-RL and SS-CCL are compared under the search
budgets of 20, 50, 100, 150, and 200. According to a search
budget, neural architectures are randomly selected from the
search space as the training dataset, and the remaining neural
architectures constitute the test dataset. To illustrate the effect
of the training epochs, the neural predictors with different
search budgets are trained under 50, 100, 150, 200, 250, and
300 training epochs. After fine-tuning, the correlation between
the validation accuracy of the neural architectures and their
performance predicted by the neural predictors is evaluated
using the Kendall tau rank correlation. All the experimental
results are averaged over 40 independent runnings using different
random seeds.
4) Results
The predictive performance measurements of the neural predictors
on NASBench-101 and NASBench-201 are shown in Figure
4 and Figure 5, respectively.
On the NASBench-101 search space, SS-RL achieves its best
performance with fewer training epochs and gradually decreases
with more training epochs, and finally drops to a range of 0.2 to
0.3. The supervised neural predictor performs significantly better
than SS-RL when the training epoch is above 150 and the
search budget is more than 100. The predictive performance of
SS-CCL is significantly better than that of SS-RL and the supervised
neural predictor. It increases with the number of training
epochs, and approaches saturation when the search budget
exceeds 150. In addition, SS-CCL achieves better performance
than the supervised neural predictor while using half of the
number of training neural architectures. In extreme cases,
SS-CCL achieves comparable performance to the supervised
neural predictor using only a quarter of the training neural architectures
(Figure 4e and Figure 4f).
On the NASBench-201 search space, SS-RL and SS-CCL
have comparable performance and both consistently outperform
the supervised neural predictor. SS-CCL slightly outperforms
SS-RL when trained for more than 150 epochs. As
shown in Figure 5b and Figure 5f, even with fewer training
epochs, SS-RL and SS-CCL can approach their optimal performance.
When trained over 200 epochs, SS-RL and
SS-CCL can outperform the supervised neural predictor using
only a quarter of the training samples (Figure 5e, Figure 5f).
Nvidia TITAN V GPUs are used to train the two self-supervised
representation learning methods and report the time-cost in
Table II. When trained on the NASBench-101 search space with
the same epochs, the time cost of self-supervised central contrastive
learning is about six times higher than that of self-supervised
regression learning. On the small search space of NASBench-201,
their training time costs are comparable and both methods are
quite efficient in training.
The above results show that the performance of SS-RL on
the small search space of NASBench-201 is comparable to that of
TABLE II Time cost of self-supervised training.
METHODS
SEARCH
SPACE
Self-Supervised
Regression
Self-Supervised Central-Contras*
Self-Supervised
Regression
Self-Supervised
Central-Contras*
101*
101*
201*
201*
TRAINING
EPOCHS
300
300
1000
300
BATCH
SIZE
-
14K
-
10K
GPU
DAYS
0.92
5.97
0.05
0.02
*NASBench-101,
NASBench-201 and Central Contrastive are shortened as 101, 201
and Central-Contras, respectively.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 41
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
