IEEE Computational Intelligence Magazine - August 2021 - 37

convolution 11 ,# and max-pooling 33 .# Figure 1a illustrates a
neural architecture in NASBench-101 search space and uses
green and red lines to indicate two different input-to-output
paths. The operations and their corresponding indices of the
neural architecture are shown in Figure 1b. Figure 1c demonstrates
two input-to-output paths of the neural architecture in
Figure 1a. Unlike path-based encoding [3], when all the inputto-output
paths are extracted from the neural architecture, the
index of operations in the input-to-output paths is also recorded.
The position-aware path-based encoding of the two different
paths in Figure 1c is indicated in Figure 1d. The vector length of
each input-to-output path is fixed, which equals the multiplication
of the number of operation nodes and the number of
operation types. All the operation nodes are traversed in the
neural architecture. If an operation node appears in the inputto-output
path, then the operation is represented by its one-hot
operation type vector; otherwise, it is represented by a zero vector.
Since there are three different operations in NASBench-101,
the length of the one-hot operation vector and the
zero vector is three.
The final encoding vector is the concatenation of the position-aware
path-based encoding vectors for all the input-to-output
paths in the neural architecture. In order to maintain the
consistency of the connection, the following operations are performed
sequentially:
1) Firstly, sort all the input-to-output paths in ascending order
by path length.
2) Secondly, sort all the input-to-output paths of the same path
length in ascending order by the operation index.
3) Finally, concatenate the sorted input-to-output paths' position-aware
path-based encoding vectors.
The vector length of path-based encoding [3] increases exponentially
with the number of operation nodes, whereas the vector
length of the position-aware path-based encoding increases linearly
with the number of input-to-output paths. Therefore, the
position-aware path-based encoding is a more efficient vector
encoding scheme than path-based encoding. As the number of
input-to-output paths may be different in different neural architectures,
the short vectors will be padded with zeros to keep all
vectors the same length.
Since only the structural information of neural architectures is
considered, the position-aware path-based encoding scheme does
not cover other important properties of neural architectures such
as input resolution and kernel numbers. However, the positionaware
path-based encoding scheme is flexible and scalable. With
some slight modifications, the current encoding scheme can
accommodate other neural architecture settings. A modified position-aware
path-based encoding scheme, including the neural
architecture's input resolution and kernel numbers, is presented in
the Supplementary Materials.
C. Self-Supervised Regression Learning
The pretext task of the proposed self-supervised regression learning
is to predict the normalized GED of the two input neural
architectures. GED is defined as
GE ,, ,,
DssssSppij
^h / k
=- !
=
ki
K
i
j
k
ij
(3)
Output
Node
[Input, Conv 3 × 3, Conv 3 × 3,
[Input,
Conv
1 × 1
Conv
3 × 3
Conv
3 × 3
Input
Node
(a)
Max-Pool
3 × 3
Input
Node
Input-to-Output
Path 1
Input
Node
Input-to-Output
Path 2
Conv
3 × 3
1,
2,
(b)
Conv
1 × 1
13
Conv
3 × 3
Conv
3 × 3
Conv
1 × 1
Max-Pool
3 × 3
12 34
(c)
FIGURE 1 Overview of the position-aware path-based encoding. (a) A neural architecture in NASBench-101 search space. The green and red
lines indicate two input-to-output paths. (b) Operations and their corresponding unique indices. (c) Two different input-to-output paths and
their operation indices. (d) Position-aware path-based encoding of the two input-to-output paths in (c).
Output
Node
Output
Node
Max-Pool
Conv 1 × 1, 3 × 3 , Output]
3,
4,
Output]
Input-to-Output Path 1 Encoding
Input-to-Output Path 2 Encoding
[1,0,0] [0,0,0] [0,1,0] [0,0,0]
[1,0,0] [1,0,0] [0,1,0] [0,0,1]
(d)
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 37

IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021