IEEE Computational Intelligence Magazine - August 2021 - 27

chosen to mutate, the mutation picks one of the other operations
based on (1). The depth mutation can change the depth
of the subnetwork in the over-parameterized network by setting
the operation of one block to the 'Identity' operation. To
search more efficiently and evaluate operations that are more
reasonable, we limit the model size as a restriction metric. The
structure can then evolve into a subnetwork that has the same
computational burden. The probability of the restriction metric
pi
rm is defined as
-
po ,
where ()oMS
kn,
i
rm () =
kn
i
/
kl
represents the number of parameters of the kth
operation in the ith block and n represents the index of epoch
or current number of evolutions. The final evolution probability
p that combines pmt
and prm
po )) () () () (),
kn
i
,, 1 po ,11 rm
=+ -
mm
po
mt
kn
i
kn
i
is defined as
(3)
where 1m is a hyperparameter.
Mutation validation: After each evolution, the subnetwork
is trained using (6), and the loss is used as the evaluation
metric. We observe the current validation loss a and accordingly
update the loss (),lo ,kn
i
tion loss of all the sampled operations o(, )
ij
k
lo mm2
() () () ,a=+ -,,-1
kn
i
2)) (4)
lo
kn
i
1
where 2m is a hyperparameter. If the operation which is mutated
performs better (less loss), we apply it as the base of the next
evolution; otherwise, we use the original operation as the base
of the next evolution.
o , = )
kn
i
ol ol o
o
kn
i
kn
i
,, ,
,
,( )( )
,
kn
i
-1 else
1
kn
i
-1
(5)
C. Contrastive Learning
Contrastive learning [37] greatly improves the performance of
unsupervised visual representation learning. The goal is to keep
the negative sample pairs away and the positive samples close in
the latent space. Prior works [1], [38] usually investigate contrastive
learning by exploring the sample pairs calculated from the
encoder and the momentum encoder [1]. Based on the investigation,
we reformulate the unsupervised/self-supervised NAS as
a teacher-student model, as shown in Fig. 2 (a). Following [1],
we build dynamic dictionaries, and the " keys " t in the dictionary
are sampled from data and are represented by the teacher network.
In general, the key representation is
i (.)( ;;.) is a teacher network, and xt
tf (),xt
= it
ffo ,kn
i
=
t
i =
s
o ,kn
i
is
it
= is
where
is a key sample.
where
Likewise, the " query " xs is represented by sf (),xs
ff (; ;.) is a student network. Unsupervised learning
performs dictionary look-up by training a network of students.
The student model and teacher model are subnetworks of NAE
from the over-parameterized network described in 3.1.
Algorithm 1 FaUNAE.
Input: Training data, validation data, and the initial structure 0a
Parameter: Searching hypergraph
Output: Optimal structure a
1: Let
n .1=
2: while ()K 1> do
3:
for tT , ...,1=
4:
5:
epoch do
Evolve architecture na from the old architecture n 1a -
based on the evolution probability p using (3);
Construct the Teacher model and Student model with
the same architecture
an , and then train Student models
by gradient descent and update Teacher model by
EMA using (7);
6:
Get the evaluation loss on the validation data using (6);
Use (4) to calculate the performance and assign it to all
sampled operations;
Update na using (5);
7: end for
8:
9:
10:
11:
if tK E)==
K K 1! - ;
then
Update ow
12: end if
13: end while
14: return a
AUGUST 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 27
kn,
^ i h using (9);
Reduce the search space: XX! - " argmax o^w
k
ii
kn
i
, h, ;
which historically indicates the validaas
exp
exp
MS
MS
()
()
o
o
kn
i
,
kn
i
l,
,
(2)
We
use contrastive loss to match an encoded query s with the
encoded key dictionary to train the visual representation student
model. The value of the contrastive loss is lower when s and t are
from the same (positive) sample and higher when s and t are
from different (negative) samples. The contrastive loss is also
deployed in FaUNAE as a measure to guide the evolution of the
structure to obtain the optimal structure based on the unlabeled
dataset. InfoNCE [39] shown in Fig.2 (a), measures the similarity
using the dot product and is used as our evaluation metric:
L =-log N
exp st$ +
n=0
where x is a temperature hyperparameter per [40] and t+
represents
the feature calculated from the same sample with s.
InfoNCE is over one positive and M negative samples. Our
method is general and can be based on other contrastive loss
functions [37], [40]-[42]. Following [1], [43], the teacher
model is updated as an exponential moving average (EMA) of
the student model:
ii i=+ -tt smm1
))
() ,
(7)
where si and ti are the weight of the student model and the
teacher model, respectively, updated by back-propagation in
contrastive learning, and
hyperparameter.
m [, )01!
is a smoothing coefficient
(/x)
,
/ exp stn x (/ )
$
(6)
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
