IEEE Computational Intelligence Magazine - August 2021 - 44

SS-CCL, but on the large search space of NASBench-101, its
performance drops significantly. We conjecture that this is because
SS-RL does not get enough paired training samples and training
time, so it is difficult to learn to generate meaningful feature representations.
This suggests that self-supervised regression learning
may not be suitable for large search space. Unlike
self-supervised regression learning, self-supervised central contrastive
learning does not need paired training samples. The
optimization purpose of central contrastive learning is more
intuitive, which pulls similar neural architectures to gather closely
in the feature space and push dissimilar neural architectures
far away. When the search space is large, we advocate the adoption
of the self-supervised central contrastive learning method,
and self-supervised regression learning can be considered when
the search space is small.
D. Effect of Batch Size
Since the number of negative pairs of central contrastive learning
is determined by the batch size, in this section, experiments are
conducted to investigate the effect of batch size on the performance
of neural predictors. The batch size N is set to 10k, 40k,
70k, and 100k, and the corresponding neural predictors are
denoted as SS-CCL-10k, SS-CCL-40k, SS-CCL-70k, and SSCCL-100k,
respectively. The number of training architectures M
is set to half of the batch size. To compare the performance of
neural predictors with larger M, SS-CCL pre-trained with
N 140k=
and M 140k=
is also included, and it is denoted as
SS-CCL-140k. The initial learning rate of the above neural predictors
is set to 5e 3,and
all the experiments in this section are
carried out on NASBench-101. Other experimental settings are
the same as in Section IV-C. All results are averaged over 40 independent
runnings using different seeds.
As shown in Figure 6, SS-CCL outperforms the supervised
neural predictor despite the batch size used. When the search
budget is greater than 100 and the training epoch is greater
than 100, the performance of SS-CCL trained with different
batch sizes tends to be the same (Figure 6c-6f). Unlike the
findings in the study of contrast learning-based image classification
[9], i.e., where model performance increases consistently
with batch size, using larger batch sizes does not improve the
performance of SS-CCL. The results demonstrate that SS-CCL
is not sensitive to batch size when the search budget is large.
Therefore, when using the self-supervised central contrastive
learning method to pre-train the neural predictor's embedding
part, a relatively small batch size can be selected to obtain better
memory efficiency.
E. Predictive Performance Comparison
The neural predictors proposed in this paper are compared with
SemiNAS [34], Multilayer Perceptron (MLP) [4], and that proposed
in Wen et al. [7] (denoted as NP-NAS). The comparison is
performed on NASBench-101, and all the predictors are trained
for 200 epochs under the search budgets of 20, 50, 100, and 200.
The experimental results are averaged over 40 independent runnings
using different random seeds.
44 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
As illustrated in Figure 7, the neural predictor SS-CCL
achieves the best performance for all of the search budgets.
The supervised neural predictor outperforms SemiNAS and
MLP when the search budget is less than 100, but it is surpassed
by MLP and NP-NAS when the search budget
is
greater than 100. Since self-supervised regression learning is
not efficient for learning meaningful representations when the
search space has numerous neural architectures, the performance
of SS-RL is only slightly better than that of SemiNAS,
which has the worst performance.
F. Fixed Budget NPENAS
1) Setup
The pre-trained neural predictors are integrated with NPENAS
[6] and the integration of SS-RL and SS-CCL with NPENAS
are denoted as NPENAS-SSRL and NPENAS-SSCCL, respectively.
The fixed budget version of NPENAS-SSRL and NPENAS-SSCCL
are denoted as NPENAS-SSRL-FIXED and
NPENAS-SSCCL-FIXED, respectively. The experimental settings
are the same as NPENAS [6]. The algorithms of random
search (RS) [50], regularized evolutionary (REA) [25], BANANAS
[3] with path-based encoding (BANANAS-PE), BANANAS
with position-aware path-based encoding (BANANAS-PAPE),
NPENAS-NP [6], and NPENAS-NP with a fixed search budget
(NPENAS-NP-FIXED) are compared to illustrate the benefit of
the two self-supervised learning methods for NAS. Each algorithm
has a search budget of 150 and 100 on the NASBench-101
and NASBench-201 search space, respectively. The fixed search
budgets for NASBench-101 and NASBench-201 are set to 90
and 50, respectively. All the experimental results are averaged over
600 independent trails, every update of the population, each algorithm
returns the architecture with the lowest validation error so
far and reports its test error, so there are 15 or 10 best architectures
in total. The methods proposed in this paper are also compared
with arc2vec [8] that is a recently proposed unsupervised
representation learning for NAS. As the search strategies employ
the neural architectures' validation error to explore the search
space, a reasonable best performance of NAS is the test error of
the neural architecture that has the best validation error in the
search space, which is called the ORACLE baseline [7]. The
ORACLE baseline is used as the upper bound of performance.
Since self-supervised central contrastive learning is more suitable
for a large search space, the experiments on the DARTS search
space are conducted only using the self-supervised central contrastive
learning method. As the DARTS search space contains 1018
neural architectures, it is not possible to train the self-supervised
central contrastive representation learning model using all of the
neural architectures. The model is trained with 50k randomly sampled
neural architectures. The training details on the DARTS search
space are the same as those on NASBench-101. After self-supervised
pre-training, NPENAS-SSCCL-FIXED is adopted to search
architectures in the DARTS search space. The NPENAS-SSCCLFIXED
has a search budget of 100 but can only evaluate 70
searched architectures. When the number of searched architectures
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
