IEEE Computational Intelligence Magazine - August 2021 - 48

TABLE VI Performance comparison of NAS algorithms on DART for CIFAR-10 Classification.
MODEL
NASNET-A [23]
NAONET [51]
ASHA [50]
DARTS [20]
BANANAS [3]
GATES [5]
ARCH2VEC-RL [8]
ARCH2VEC-BO [8]
NPENAS-NP [6]
NPENAS-SSCCL-FIXED
PARAMS (M)
3.3
10.6
2.2
3.3
3.6
4.1
3.3
3.6
3.5
3.9
ERR(%) AVG
-
-
-
-
2.64 ± 0.05
-
2.65 ± 0.05
2.56 ± 0.05
2.54 ± 0.10
2.49 ± 0.06
that the neural predictor SS-CCL outperforms SS-RL on the
large search space of NASBench-101.
3) NAS Results on NASBench-201
The above algorithms are compared on CIFAR-10, CIFAR-100,
and Image Net-16-120 on NASBench-201, and the results are
shown in Figure 9, Figure 10 and Figure 11, respectively. The
quantitative comparison is presented in Table V. As arc2vec [8] does
not report queries on this benchmark, it is not compared here.
Table V shows that the methods proposed in this paper obtain
the best performance on all three datasets. Specifically, NPENASSSCCL
achieves the best performance on both CIFAR-100 and
ImageNet-16-120, almost reaching the ORACLE baseline on
CIFAR-100 (26.5% vs. 26.49%). In particular, the performance
of NPENAS-SSCCL is the same as the ORACLE baseline on
ImageNet-16-120. On CIFAR-10, NPENAS-SSRL-FIXED
achieves the best performance comparable to the ORACLE
baseline (8.92% vs 8.91%). In addition, like the results on
sep_conv_3×3
c_{k-2}
skip_connect
sep_conv_5×5
c_{k-1}
sep_conv_3×3
sep_conv_3×3
none
c_{k-1}
c_{k-2}
none
max_pool_3×3
(b)
avg_pool_3×3
max_pool_3×3
sep_conv_5×5
1
sep_conv_3×3
1
sep_conv_3×3
(a)
dil_conv_5×5
ERR(%) BEST
2.65
3.18
2.85
3.00 ± 0.14
2.57
2.58
2.60
2.48
2.44
2.41
NO. OF SAMPLES EVALUATED
20000
1000
700
-
100
800
100
100
100
70
GPU DAYS
1800
200
9
1.5
11.8
-
9.5
10.5
1.8
1.6
NASBench-101, the performance of BANANAS using the position-aware
path-based encoding exceeds that of using path-based
encoding. Furthermore, it can be seen from Figures 9-11 that the
performance of our methods improves faster as the search budget
increases. Due to space limitation, more comparisons are attached
in the Supplementary Materials.
4) NAS Results on DARTS
As shown in Table VI, NPENAS-SSCCL-FIXED achieves the
best performance compared with the recently proposed NAS
algorithms, and its search speed is nearly the same as the gradientbased
method DARTS [20]. The searched normal cell and reduction
cell are illustrated in Figure 12.
sep_conv_5×5
max_pool_3×3
FIGURE 12 (a) The normal cell and (b) reduction cell searched by NPENAS-SSCCL-FIXED on DARTS.
V. Conclusion
This paper presents a new neural architecture encoding scheme,
position-aware path-based encoding, to calculate the GED of
neural architectures. To enhance the performance of neural
predictors, two self-supervised learning
methods are proposed to pre-train
the neural predictors' architecture
embedding modules to generate
meaningful representation of neural
architectures. Extensive experiments
demonstrate the superiority of the
self-supervised pre-training. The
results advocate the adoption of the
self-supervised central contrastive representation
learning method, while
self-supervised regression learning
can be considered when the search
space is small. When integrating the
pre-trained neural predictors with
NPENAS, it achieves state-of-the-art
performance on the NASBench-101,
NASBench-201 and DARTS search
space. Since neural predictors can be
combined with different search strategies,
the proposed self-supervised
representation learning methods are
2
c_{k}
3
3
2
c_{k}
48 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
