IEEE Computational Intelligence Magazine - August 2021 - 35

After self-supervised pre-training, two neural predictors are
built by connecting a fully connected layer to the architecture
embedding modules of the pre-trained models. Finally, the pretrained
neural predictors are respectively integrated into a neural
predictor guided evolutionary neural architecture search (NPENAS)
algorithm [6] to verify their performance.
Our main contributions can be summarized as follows.
❏ A new position-aware path-based neural architecture encoding
scheme is devised to overcome the drawbacks of adjacency
matrix encoding and path-based encoding methods. The
experimental results illustrate its superiority in identifying
unique neural architectures.
❏ A self-supervised regression learning method is proposed,
which defines a pretext task to predict the normalized GED
of two different neural architectures and design a graph neural
network-based model with two independent identical
branches to learn meaningful representation of neural architectures.
The neural predictor pre-trained by this method
achieves its performance upper bound with a small search
budget and a few training epochs. In the best case, it can
achieve better performance only using half of the search budget
compared to its supervised counterparts.
❏ A self-supervised central contrastive learning algorithm is proposed,
which forces neural architectures with small GED to
lean closer together in the feature space, while neural architectures
with large GED are divided further apart. The pre-trained
neural predictor fine-tuned with a quarter of the search budget
can achieve comparable performance to its supervised counterparts;
with the same search budget, the fine-tuned neural predictor
outperforms its supervised counterparts by about 1.5
times. The proposed central contrastive learning algorithm can
also be extended to the domain of graph unsupervised representation
learning without any modifications.
❏ Incorporating the pre-trained neural predictors, NPENAS
achieves state-of-the-art performance on NASBench-101 [17],
NASBench-201 [19], and DARTS [20] benchmarks. On
NASBench-101 and NASBench-201 search space, the
searched neural architectures even achieve comparable or equal
results to the ORACLE baseline (performance upper bound).
II. Related Work
A. Neural Architecture Search
Due to the huge size of the pre-defined search space, NAS usually
searches for potential superiority neural network architectures by
utilizing a search strategy. Reinforcement learning (RL) [2], [21]-
[23], evolutionary algorithms [6], [24]-[28], gradient-based methods
[20], [29]-[32], and Bayesian optimization (BO) [3], [6], [33]
are the commonly used search strategies. A search strategy adjusts
itself by exploiting the selected neural architectures' performance
metrics to explore the search space better.
As it is time-consuming to estimate the performance metrics
of a given neural architecture through training and validation procedures,
many performance estimation strategies are proposed to
speed up this task. Commonly used strategies include using a
proxy dataset and proxy architecture, early stopping, inheriting
weights from a trained architecture, and weight sharing [1]. A
neural predictor that is employed to estimate the performance
metrics of the neural network architectures can also be recognized
as a kind of performance estimation strategy. Recently,
many search strategies have adopted neural predictors to explore
the search space [2]-[4], [6]-[8], [34], [35]. The capability of neural
predictors to accurately predict the performance of neural architectures
is critical to the search strategies using neural predictors.
The neural predictors are trained on a dataset consisting of a
number of neural architectures, along with corresponding performance
metrics, which are obtained through time-consuming
training and validation procedures. Wen et al. [7] designed an
architecture encoding method and proposed a neural predictor
composed of Graph Convolutional Networks (GCNs). BRPNAS
[35] adopts GCNs to construct a latency neural predictor
and employs transfer learning to transfer knowledge from the
trained latency predictor to a binary relation predictor, which is
used to rank neural architectures. GATES [5] views the embedding
of neural architectures as the information flow in the architectures
and presents two different encoding processes: operation
on node and operation on edge. Each operation node employs a
soft attention mask to enhance the input features, and the output
of the output node is used as the embedding of the neural architecture.
SemiNAS [34] presents a semi-supervised iterative training
scheme to reduce the number of architecture-accuracy data
pairs required to train a high-performance neural predictor. SemiNAS
first trains the neural predictor with a small number of
architecture-accuracy data pairs, then utilizes the trained neural
predictor to predict the performance of a large number of architectures,
and finally adds the generated pseudo data pairs to the
original training set to update the neural predictor.
In this paper, self-supervised representation learning is applied
to the NAS domain. Two self-supervised representation learning
methods are proposed to improve the feature representation of
neural predictors, thus enhancing the prediction performance of
neural predictors.
B. Neural Architecture Encoding Scheme
Neural architecture is usually defined as a direct acyclic graph
(DAG). The adjacency matrix of the graph is used to represent the
connections of operations, and the nodes are used to represent the
operations. The commonly used neural architecture encoding
schemes can be categorized into the vector encoding scheme and
graph encoding scheme.
Adjacency matrix encoding [4], [29], [36] and path-based
encoding [3] are two frequently used vector encoding schemes.
The adjacency matrix encoding is the concatenation of the flattened
adjacency matrix and the one-hot encoding vector of each
node, but it cannot identify the isomorphic graphs [8]. Path-based
encoding is the encoding of the input-to-output paths of the
neural architecture, but as demonstrated in the Supplementary
Materials1, this scheme cannot recognize the position of
1The supplementary material is available at https://github.com/auroua/SSNENAS.
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 35
https://www.github.com/auroua/SSNENAS
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
