IEEE Computational Intelligence Magazine - August 2021 - 36

operations in the neural architecture. The graph encoding scheme
represents the neural architecture by its adjacency matrix and the
one-hot encoding of each node.
In this paper, a new vector encoding scheme denoted as position-aware
path-based encoding is proposed. This encoding
scheme can identify different graphs more efficiently and recognize
the position of operations in the neural architecture. A graph
neural network is adopted to embed the neural architectures into
feature space. Since the graph encoding scheme cannot identify
isomorphic graphs [8], position-aware path-based encoding is
used to filter out isomorphic graphs. It is also utilized to calculate
the GED of different neural architectures.
C. Unsupervised Representation Learning for NAS
Unsupervised representation learning methods fall into two categories:
generative and discriminative [9]. The learning objective
of existing generative unsupervised learning methods for NAS,
arc2vec [8], and NASGEM [14] is to reconstruct the input neural
architectures using an encoder-decoder network, which has
little relevance to NAS. Moreover, arc2vec [8] adopts the variational
autoencoder [37] to embed the input neural architectures
into a high dimensional continuous feature space, and the feature
space is assumed to follow Gaussian distribution. Since there is
no guarantee that the real underlying distribution of the feature
space is Gaussian, this assumption may harm the representation
of neural architectures. NASGEM [14] adds a similarity loss to
improve the feature representation. However, the similarity loss
only considers the adjacency matrix of the input neural architecture
and ignores the node operations, resulting in the failure to
identify isomorphic graphs.
In this paper, two self-supervised learning methods are proposed
for NAS. The first one is inspired by unsupervised graph
representation learning. GMNs [38] adopts the graph neural network
as building blocks and presents a cross-graph attentionbased
mechanism to predict the similarity of the two input
graphs. SimGNN [39] takes two graphs as input, embeds the
graph and each node of the graph into the feature space using a
graph convolutional neural network, and then uses graph feature
similarity and node feature similarity to predict the similarity of
the input graphs. UGRAPHEMB [40] takes two graphs as input,
adopts the graph isomorphism network (GIN) [41] to embed the
input graphs into feature space, and utilizes a multi-scale node
attention mechanism to predict the similarity of the input graphs.
Our work is similar to UGRAPHEMB, but it designs a new
neural network model without using complex multi-scale node
attention and applies unsupervised learning to the field of neural
architecture representation learning.
The second method is inspired by contrastive learning for
image classification. The contrastive learning used in image classification
forces the image and its transformations to be similar in
the feature space [9], [10], [42]. Since there is no guarantee that a
neural architecture and its transformed form will have the same
performance metrics, it is not reasonable to directly apply contrastive
learning for image classification to the NAS domain. This
paper proposes
a new contrastive learning algorithm,
36 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
self-supervised central contrastive learning, to learn meaningful
representation of neural architectures. To the best of our knowledge,
this is the first study applying contrastive learning to the
NAS domain.
III. Methodology
To enhance the prediction performance of neural predictors, two
self-supervised representation learning methods are proposed to
improve the feature representation ability of the neural predictors.
A new neural architecture encoding scheme is designed to calculate
the GED of graphs in Section III-B. The self-supervised
regression learning that utilizes a carefully designed model with
two independent identical graph neural network branches to predict
the GED of neural architectures is discussed in Section III-C.
The self-supervised central contrastive learning is introduced in
Section III-D. The utilization of the pre-trained neural predictors
for the downstream search strategies is elaborated in Section III-E.
A. Problem Formulation
In a pre-defined search space S, a neural architecture s can be represented
as a DAG
sVEs S! ,
=^h
where Vvi iH:1
in s, Ev , v ,ij H1
= " , =
=
lated as
^h
yf ,s=t
(2)
where f is the neural predictor, and it takes a neural architecture s
as input and predicts the performance metric yt of s.
B. Position-Aware Path-Based Encoding
Since the proposed self-supervised learning methods utilize GED
to measure the similarity of different neural architectures, it is critical
to calculate GED effectively. This paper presents a new vector
encoding scheme, position-aware path-based encoding, which
improves path-based encoding [3] by recording the position of
each operation in the path. The scheme consists of two steps: generating
the position-aware path-based encoding vectors for the
input-to-output paths of the neural architecture, and concatenating
the vectors of all of the paths.
As shown in Eq. 1, a neural architecture can be defined by
DAG with its nodes representing the operations in the neural
architecture. DAG consists of an input node, some operation
nodes, and an output node connected in sequence. The adjacency
matrix of DAG is used to represent the connections of the different
nodes. Since each node in DAG has a fixed position, each
node is assigned with a unique index, which implies that each
operation associated with the node has a unique index.
NASBench-101 [17] is a widely used NAS search space. It
contains three different operations: convolution
33 ,#
,,
(1)
is the set of nodes representing operations
= " ij ,: is the set of edges describing the connection
of operations, and H is the number of nodes.
The prediction process of neural predictor can be formu
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
