IEEE Computational Intelligence Magazine - August 2021 - 39

where M is the number of central vectors, and ei
central vectors.
and ej
are the
After the self-supervised central contrastive pre-training, a
fully connected layer is appended to the architecture embedding
module to predict the input neural architecture's performance.
The regression loss in Eq. 6 is adopted to fine-tune the
neural predictor.
E. Fixed Budget NPENAS
NPENAS [6] combines the evolutionary search strategy with a
neural predictor and utilizes the neural predictor to guide the
evolutionary search strategy to explore the search space. In this
Algorithm 1 Central Contrastive Learning.
1: Input: batch size N, number of training architectures M and
MN ,# temperature ,x regularization weight ,m model
f .ccl
2: for sampled minibatch Ss 1
3: Ec
4:
bk k
N
=Q
for alltM1 f! " ,, , do
Randomly draw one neural architecture sS .ib
6: gGED ss where =- +
# GED: Eq. 3
5:
min =min
7:
8:
9:
10:
12:
14:
15:
ij
Epos =Q and Eneg
jj
=Q
for allji iN11 1
e fs= ccl
11: EE ej
else
negneg
! ,
end for
16: EE ei
17:
pos ! pos ,
ee1
e
c =
E
pos
! ,
18: EE ecc c
19:
20:
21:
=Q
23:
24:
26:
27:
28:
for all idx, e Ep
Epair
22: EE sim ,pc
for all e En
!
25: EE sim ,cn
end for
lt,idx =-log
29: ll,ttidx
idx
end for
=/
30: end for
31:
L
t
=+m reg
=
1
32: Update model fccl
33: end for
34: return model fccl
1 lL Ec
M / t
M
() # Lreg: Eq. 7
to minimize L.
simE
vecpair
sim ee ee h
pair ! pair ,
nc, nc ncx
=
<
/^
exp()
simpc
,
/ exp ()
!
simvec
5:
neg
sim ee ee h
pair ! pair ,
pc, pc pcx
=
<
/^
do
) / # vector average
!Epos
! pos do # idx is the index of ep in Epos
if GE (, )Ds sgmin
pos ! pos ,
ij ==
13: EE ej
end
then
! f -+ " ,, ,, , do
()
!
(, ) ji iN11 1ff ,
" ,, ,, ,
A. Benchmark Datasets
All experiments are performed on NASBench-101 [17], NASBench-201
[19], and DARTS [32] search spaces.
a) NASBench-101
NASBench-101 [17] contains 423k neural architectures, and each
architecture is trained three times on the CIFAR-10 [46] training
datasets independently. The structure of the neural architectures, as
well as their validation accuracies, and test accuracies corresponding
to the three independent trainings on CIFAR-10 are reported.
The architecture in this search space is defined by DAG, utilizing
nodes to represent the operations of the neural architecture and
Algorithm 2 Fixed Budget NPENAS.
1: Input: initial population size
= f 0
12
,, ,,
=
= " , do
paper, the pre-trained neural predictors are integrated respectively
with NPENAS to illustrate the performance gains that result
from applying self-supervised learning to NAS.
Since our experiments demonstrate that the neural predictor
built from a self-supervised pre-trained model can significantly
outperform its supervised counterpart and achieve comparable
performance with smaller training datasets, the NPENAS method
is modified to utilize only a fixed search budget to carry out the
neural architecture search. The fixed budget NPENAS is summarized
in Algorithm 2, which is modified from NPENAS and only
the differences are presented.
IV. Experiments and Analysis
In this section, experiments are conducted to illustrate that the
performance of the proposed neural predictors can be significantly
improved by self-supervised learning. The benefit of self-supervised
learning for NAS is also demonstrated by integrating the
pre-trained neural predictors with NPENAS.
All experiments are implemented in Pytorch [43]. GIN is
implemented using the publicly available graph neural network
library pytorch_geometric [44]. The code of this paper is provided
in reference [45].
n ,0 initial population Ds yii
in , neural predictor f, number of the total
searched architectures total_num, number of the evaluated
architectures (budget) to fine-tune neural predictor ft_num.
2: for n from n0
3:
4:
if nf _t num#
to total_num do
then
Initialize the weights of neural predictor f with the
weights from the pre-trained model.
Fine-tune the neural predictor f with dataset Ds yii
in,
=12 f ,, ,.
6: end
7:
Utilize the neural predictor f to guide the evolutionary neural
architecture search. # Detailed code can be found in
Algorithm 2 of NPENAS [6].
8: end for
9: Output:
sy sy Dii i
) = argmin (),( ,)!
= " (, ),
= " (, ),
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 39
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
