IEEE Computational Intelligence Magazine - August 2021 - 38

where pi
and pj
vectors of architecture si
are the position-aware path-based encoding
k
and s ,j pi
and pj
k
of the position-aware path-based encoding vector pi
are the kth elements
and p ,j
K is the vector length.
Following [39], the normalized GED is defined as
nGED ss h
ij
^ ==-exp dist
,, where
dist
GEDs s h
V
^ ij
,
,
(4)
where V is the number of nodes in the neural architectures
and exp represents the exponential function with base e. The
performance of normalized GED and non-normalized GED
are compared on NASBench-201 and the result shows that
the normalized GED performs slightly better than non-normalized
GED.
As the architecture in search space is represented as a DAG, it
is straightforward to adopt graph neural networks to aggregate
features for each node and generate the graph embedding by
averaging the nodes' features. In this paper, both the self-supervised
models and neural predictors utilize the spatial-based graph
neural network GIN layers.
Since the pretext task is to predict the normalized GED of
two different neural architectures, a regression model frl
consisting
of two independent identical graph neural network branches
is designed, as shown in Figure 2. Each branch is composed of
three sequentially connected GIN layers and a global mean pooling
(GMP) layer. The GMP layer outputs the mean of the node
features of the last GIN layer. The outputs of the two branches are
concatenated, and then sent to two sequentially connected fully
connected layers to predict the two input architectures' normalized
GED. The regression loss function to optimize the parameters
wrl
of frl
rl
is formulated as
wfss nGED ss
(, )
)=!
rl
ij
wrl
ij
/ ^^ hh^ ij h2
argmin ,, .
ss S
(5)
D. Self-Supervised Central Contrastive Learning
This paper proposes a central contrastive learning algorithm to
force neural architectures with a small GED to lean closer together
in the feature space, while neural architectures with a large
GED are divided further apart.
As illustrated in Figure 3, a graph neural network model f lcc
is
developed to embed the neural architecture into feature space.
Following SimCLR [9], f lcc
consists of a neural architecture
embedding module and two fully connected layers. For a fair
comparison, the architecture embedding module is identical to
that of
f .rl
Given a batch of neural architectures Ss N
ral architecture sS ,ib
! the minimum GED of si
Architecture
Embedding
Input
Performance
Predictor
pred
Input
Architecture
Embedding
FIGURE 2 Structure of the regression model frl.
together with ,si
bk k=1
= " , and a neufrom
all other
architectures is denoted as g .min The neural architectures whose
GED from si is equal to g ,min
positive sample set Spos
constitute the
this batch but not in Spos is denoted as S .neg The model f lcc
and Sneg
are denoted as Epos
and E ,neg
ly. A central vector ec
of s .i The set of neural architectures in
is
used to embed all of the neural architectures, and the feature vector
sets of Spos
respectiveis
calculated as the average of all of the
feature vectors in E .pos The contrastive loss is used to aggregate all
of the feature vectors in Epos
the feature vectors in Eneg
to the central vector ec
far away from .ec
and push
The detailed procedure
of central contrastive learning is summarized in Algorithm 1.
An example of the central contrastive learning is illustrated in the
Supplementary Materials.
To reduce the interaction between the central vectors, a cenArchitecture
Embedding
Input
Embedding
Feature
FIGURE 3 Structure of the feature embedding model fccl.
tral vector regularization term is added to the loss function that
forces each pair of the central vectors to be orthogonal. The central
vector regularization term is defined as
L
reg = / / 1[]eeji i
<
2
1
38 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
!
i =0 j=0
M M
j
,
(7)
and
After the self-supervised pre-training, any branch of frl
can
be selected to embed the neural architectures into the feature
space. A neural predictor is constructed by connecting a fully
connected layer to the architecture embedding module (as
illustrated in the red rectangle of Figure 2) of the pre-trained
models. Regression loss is employed to fine-tune the neural
predictor. The parameters of the neural predictor, denoted as w,
are optimized as
wfsy/ ^^ hh2
sSi
)=!
argmin
w
where
yi
is the performance metric of s .i
The structure of self-supervised regression learning method is
similar to the binary relation predictor proposed in BRP-NAS
[35]. However, the binary relation predictor requires the performance
metrics of neural architectures to learn how to rank
architectures, and the performance metrics are very time-consuming
to obtain. The self-supervised regression learning method
uses GED as supervision to learn a meaningful representation of
the neural architectures, and the time-cost of computing GED
can be neglected.
ii
,
(6)
GIN
GIN
GIN
GMP
FC
FC
GIN
GIN
GIN
GMP
FC
FC
FC
GIN
GIN
GIN
GMP
FC
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
