IEEE Computational Intelligence Magazine - August 2021 - 39
where M is the number of central vectors, and ei
central vectors.
and ej
are the
After the self-supervised central contrastive pre-training, a
fully connected layer is appended to the architecture embedding
module to predict the input neural architecture's performance.
The regression loss in Eq. 6 is adopted to fine-tune the
neural predictor.
E. Fixed Budget NPENAS
NPENAS [6] combines the evolutionary search strategy with a
neural predictor and utilizes the neural predictor to guide the
evolutionary search strategy to explore the search space. In this
Algorithm 1 Central Contrastive Learning.
1: Input: batch size N, number of training architectures M and
MN ,# temperature ,x regularization weight ,m model
f .ccl
2: for sampled minibatch Ss 1
3: Ec
4:
bk k
N
=Q
for alltM1 f! " ,, , do
Randomly draw one neural architecture sS .ib
6: gGED ss where =- +
# GED: Eq. 3
5:
min =min
7:
8:
9:
10:
12:
14:
15:
ij
Epos =Q and Eneg
jj
=Q
for allji iN11 1
e fs= ccl
11: EE ej
else
negneg
! ,
end for
16: EE ei
17:
pos ! pos ,
ee1
e
c =
E
pos
! ,
18: EE ecc c
19:
20:
21:
=Q
23:
24:
26:
27:
28:
for all idx, e Ep
Epair
22: EE sim ,pc
for all e En
!
25: EE sim ,cn
end for
lt,idx =-log
29: ll,ttidx
idx
end for
=/
30: end for
31:
L
t
=+m reg
=
1
32: Update model fccl
33: end for
34: return model fccl
1 lL Ec
M / t
M
() # Lreg: Eq. 7
to minimize L.
simE
vecpair
sim ee ee h
pair ! pair ,
nc, nc ncx
=
<
/^
exp()
simpc
,
/ exp ()
!
simvec
5:
neg
sim ee ee h
pair ! pair ,
pc, pc pcx
=
<
/^
do
) / # vector average
!Epos
! pos do # idx is the index of ep in Epos
if GE (, )Ds sgmin
pos ! pos ,
ij ==
13: EE ej
end
then
! f -+ " ,, ,, , do
()
!
(, ) ji iN11 1ff ,
" ,, ,, ,
A. Benchmark Datasets
All experiments are performed on NASBench-101 [17], NASBench-201
[19], and DARTS [32] search spaces.
a) NASBench-101
NASBench-101 [17] contains 423k neural architectures, and each
architecture is trained three times on the CIFAR-10 [46] training
datasets independently. The structure of the neural architectures, as
well as their validation accuracies, and test accuracies corresponding
to the three independent trainings on CIFAR-10 are reported.
The architecture in this search space is defined by DAG, utilizing
nodes to represent the operations of the neural architecture and
Algorithm 2 Fixed Budget NPENAS.
1: Input: initial population size
= f 0
12
,, ,,
=
= " , do
paper, the pre-trained neural predictors are integrated respectively
with NPENAS to illustrate the performance gains that result
from applying self-supervised learning to NAS.
Since our experiments demonstrate that the neural predictor
built from a self-supervised pre-trained model can significantly
outperform its supervised counterpart and achieve comparable
performance with smaller training datasets, the NPENAS method
is modified to utilize only a fixed search budget to carry out the
neural architecture search. The fixed budget NPENAS is summarized
in Algorithm 2, which is modified from NPENAS and only
the differences are presented.
IV. Experiments and Analysis
In this section, experiments are conducted to illustrate that the
performance of the proposed neural predictors can be significantly
improved by self-supervised learning. The benefit of self-supervised
learning for NAS is also demonstrated by integrating the
pre-trained neural predictors with NPENAS.
All experiments are implemented in Pytorch [43]. GIN is
implemented using the publicly available graph neural network
library pytorch_geometric [44]. The code of this paper is provided
in reference [45].
n ,0 initial population Ds yii
in , neural predictor f, number of the total
searched architectures total_num, number of the evaluated
architectures (budget) to fine-tune neural predictor ft_num.
2: for n from n0
3:
4:
if nf _t num#
to total_num do
then
Initialize the weights of neural predictor f with the
weights from the pre-trained model.
Fine-tune the neural predictor f with dataset Ds yii
in,
=12 f ,, ,.
6: end
7:
Utilize the neural predictor f to guide the evolutionary neural
architecture search. # Detailed code can be found in
Algorithm 2 of NPENAS [6].
8: end for
9: Output:
sy sy Dii i
) = argmin (),( ,)!
= " (, ),
= " (, ),
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 39
IEEE Computational Intelligence Magazine - August 2021
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
Contents
IEEE Computational Intelligence Magazine - August 2021 - Cover1
IEEE Computational Intelligence Magazine - August 2021 - Cover2
IEEE Computational Intelligence Magazine - August 2021 - Contents
IEEE Computational Intelligence Magazine - August 2021 - 2
IEEE Computational Intelligence Magazine - August 2021 - 3
IEEE Computational Intelligence Magazine - August 2021 - 4
IEEE Computational Intelligence Magazine - August 2021 - 5
IEEE Computational Intelligence Magazine - August 2021 - 6
IEEE Computational Intelligence Magazine - August 2021 - 7
IEEE Computational Intelligence Magazine - August 2021 - 8
IEEE Computational Intelligence Magazine - August 2021 - 9
IEEE Computational Intelligence Magazine - August 2021 - 10
IEEE Computational Intelligence Magazine - August 2021 - 11
IEEE Computational Intelligence Magazine - August 2021 - 12
IEEE Computational Intelligence Magazine - August 2021 - 13
IEEE Computational Intelligence Magazine - August 2021 - 14
IEEE Computational Intelligence Magazine - August 2021 - 15
IEEE Computational Intelligence Magazine - August 2021 - 16
IEEE Computational Intelligence Magazine - August 2021 - 17
IEEE Computational Intelligence Magazine - August 2021 - 18
IEEE Computational Intelligence Magazine - August 2021 - 19
IEEE Computational Intelligence Magazine - August 2021 - 20
IEEE Computational Intelligence Magazine - August 2021 - 21
IEEE Computational Intelligence Magazine - August 2021 - 22
IEEE Computational Intelligence Magazine - August 2021 - 23
IEEE Computational Intelligence Magazine - August 2021 - 24
IEEE Computational Intelligence Magazine - August 2021 - 25
IEEE Computational Intelligence Magazine - August 2021 - 26
IEEE Computational Intelligence Magazine - August 2021 - 27
IEEE Computational Intelligence Magazine - August 2021 - 28
IEEE Computational Intelligence Magazine - August 2021 - 29
IEEE Computational Intelligence Magazine - August 2021 - 30
IEEE Computational Intelligence Magazine - August 2021 - 31
IEEE Computational Intelligence Magazine - August 2021 - 32
IEEE Computational Intelligence Magazine - August 2021 - 33
IEEE Computational Intelligence Magazine - August 2021 - 34
IEEE Computational Intelligence Magazine - August 2021 - 35
IEEE Computational Intelligence Magazine - August 2021 - 36
IEEE Computational Intelligence Magazine - August 2021 - 37
IEEE Computational Intelligence Magazine - August 2021 - 38
IEEE Computational Intelligence Magazine - August 2021 - 39
IEEE Computational Intelligence Magazine - August 2021 - 40
IEEE Computational Intelligence Magazine - August 2021 - 41
IEEE Computational Intelligence Magazine - August 2021 - 42
IEEE Computational Intelligence Magazine - August 2021 - 43
IEEE Computational Intelligence Magazine - August 2021 - 44
IEEE Computational Intelligence Magazine - August 2021 - 45
IEEE Computational Intelligence Magazine - August 2021 - 46
IEEE Computational Intelligence Magazine - August 2021 - 47
IEEE Computational Intelligence Magazine - August 2021 - 48
IEEE Computational Intelligence Magazine - August 2021 - 49
IEEE Computational Intelligence Magazine - August 2021 - 50
IEEE Computational Intelligence Magazine - August 2021 - 51
IEEE Computational Intelligence Magazine - August 2021 - 52
IEEE Computational Intelligence Magazine - August 2021 - 53
IEEE Computational Intelligence Magazine - August 2021 - 54
IEEE Computational Intelligence Magazine - August 2021 - 55
IEEE Computational Intelligence Magazine - August 2021 - 56
IEEE Computational Intelligence Magazine - August 2021 - 57
IEEE Computational Intelligence Magazine - August 2021 - 58
IEEE Computational Intelligence Magazine - August 2021 - 59
IEEE Computational Intelligence Magazine - August 2021 - 60
IEEE Computational Intelligence Magazine - August 2021 - 61
IEEE Computational Intelligence Magazine - August 2021 - 62
IEEE Computational Intelligence Magazine - August 2021 - 63
IEEE Computational Intelligence Magazine - August 2021 - 64
IEEE Computational Intelligence Magazine - August 2021 - 65
IEEE Computational Intelligence Magazine - August 2021 - 66
IEEE Computational Intelligence Magazine - August 2021 - 67
IEEE Computational Intelligence Magazine - August 2021 - 68
IEEE Computational Intelligence Magazine - August 2021 - 69
IEEE Computational Intelligence Magazine - August 2021 - 70
IEEE Computational Intelligence Magazine - August 2021 - 71
IEEE Computational Intelligence Magazine - August 2021 - 72
IEEE Computational Intelligence Magazine - August 2021 - 73
IEEE Computational Intelligence Magazine - August 2021 - 74
IEEE Computational Intelligence Magazine - August 2021 - 75
IEEE Computational Intelligence Magazine - August 2021 - 76
IEEE Computational Intelligence Magazine - August 2021 - 77
IEEE Computational Intelligence Magazine - August 2021 - 78
IEEE Computational Intelligence Magazine - August 2021 - 79
IEEE Computational Intelligence Magazine - August 2021 - 80
IEEE Computational Intelligence Magazine - August 2021 - 81
IEEE Computational Intelligence Magazine - August 2021 - 82
IEEE Computational Intelligence Magazine - August 2021 - 83
IEEE Computational Intelligence Magazine - August 2021 - 84
IEEE Computational Intelligence Magazine - August 2021 - 85
IEEE Computational Intelligence Magazine - August 2021 - 86
IEEE Computational Intelligence Magazine - August 2021 - 87
IEEE Computational Intelligence Magazine - August 2021 - 88
IEEE Computational Intelligence Magazine - August 2021 - 89
IEEE Computational Intelligence Magazine - August 2021 - 90
IEEE Computational Intelligence Magazine - August 2021 - 91
IEEE Computational Intelligence Magazine - August 2021 - 92
IEEE Computational Intelligence Magazine - August 2021 - 93
IEEE Computational Intelligence Magazine - August 2021 - 94
IEEE Computational Intelligence Magazine - August 2021 - 95
IEEE Computational Intelligence Magazine - August 2021 - 96
IEEE Computational Intelligence Magazine - August 2021 - Cover3
IEEE Computational Intelligence Magazine - August 2021 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com