IEEE Computational Intelligence Magazine - August 2021 - 24

relative location prediction [7] and rotation prediction [8]. Large
scale experiments on these pretext tasks [9] reveal that neither is
the ranking of architectures consistent across different methods
nor is the ranking of methods consistent across architectures. Thus,
pretext tasks for self-supervised learning should be considered in
conjunction with their underlying architectures.
Self-supervised representation learning can also be formulated
as learning an embedding where the semantically similar
images are very close, and the semantically different images are
far away. One method is to contrast positive pairs with negative
pairs. Contrastive learning [10] learns mappings that are invariant
to certain transformations of the inputs. He, et al. [1] build a
dynamic dictionary with a queue and a moving-averaged
encoder using a contrastive loss. He, et al. [1] update the dictionary
in the form of a queue, decoupling the size of the dictionary
and the batch size, so that the size of the dictionary
does not need to be constrained by batch size. In addition, the
Momentum-based Moving Average keeps the encoder updated
all the time. Subsequently, Chen, et al. [11] combine the methods
of [1], [12], and use an MLP projection head and more
data Augmentation to further improve performance.
B. Neural Architecture Search
With the rapid development of computer vision, various network
structures have achieved significant performance improvements,
but most of them are manually designed. Recently, a
method called NAS has attracted more and more attention,
which can automate neural net architecture design [2], [13].
NAS performs full-fledged training and validation over thousands
of architectures from scratch. Traversing all candidate
architectures in a very large search space requires tremendous
computation and memory resources. A cell-based architecture
[3] is widely used to reduce the size of search space by stacking
cells to make NAS more efficient. However, the search space is
still noncontinuous and high-dimensional. A one-shot architecture
search method [14], [15] is proposed for learning efficiency,
making it possible to identify an optimal architecture within a
few GPU days. Pham, et al. [15], share parameters among child
models and trains a controller to search the network architecture.
Liu, et al. [14] introduce a differentiable framework that
enables NAS to use gradient descent for search. Due to the
considerable memory requirement of [14], [16] proposes a partial
channel connection method, which reduces memory consumption
by sampling a small part of the network and improves
search efficiency. Chen, et al. [17] successfully apply NAS to
model defense, sampling the network through the combination
of valid accuracy and bandit, and achieve good results in adversarial
training. The use of NAS for unsupervised learning is a
new research hotspot, which needs further exploration.
C. Object Detection and Segmentation
Object detection [18], [19] is one of the basic tasks of computer
vision. Object detection algorithms typically include onestage
methods and two-stage methods. The two-stage methods
use a sliding window detector. First, the algorithm generates a
24 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2021
series of anchors as samples and then classifies the samples
through the convolutional neural network. Common algorithms
include R-CNN [20], Fast R-CNN [21], and Faster
R-CNN [22]. The one-stage methods use a single shot detector,
and the methods are divided into anchor-based method and
anchor-free methods. The anchor-free method does not generate
anchors and directly transforms the problem of anchor positioning
into regression. Common algorithms are YOLO [23], SSD [24]
and so on. Compared with object detection, instance segmentation
requires more stringent requirements. It not only requires segmenting
the object's mask and identifying its category but also
needs to distinguish different instances in the same category. The
main algorithms include Mask R-CNN [25], SOLO [26], and
SOLOv2 [27]. For object detection and segmentation, we mainly
use Faster R-CNN and Mask R-CNN. Faster R-CNN proposes
the Region Proposal Network on the basis of Fast R-CNN and
integrates feature extraction, region proposal, bounding box
regression (rect refine), and classification into one network, thereby
effectively improving detection accuracy and detection efficiency.
Mask R-CNN is a framework for instance segmentation, which
adds a branch to Faster R-CNN for segmentation. In addition,
Mask R-CNN introduces a region of interest (ROI) alignment to
replace ROI pooling in Faster R-CNN, which improves the
accuracy of the mask.
D. Neural Architecture Evolution
Before the advent of the term " deep learning, " neuroevolution
[28], [29] used genetic algorithms to change the topology along
with hyperparameters and connection weights of artificial neural
networks (ANNs) by allowing high performing structures to
reproduce and mutate. Recent works focus on evolving deep
neural architecture on a larger scale. Real, et al. [13] start small
and mutates architectures to navigate large search spaces.
Encouraged by the idea of stacking the same modules in many
successful architectures such as ResNet and DenseNet, a variant
of this method [30] shows better results. It repeatedly evolves
small neural network modules by using a larger hand-coded
blueprint. Similarly, CoDeepNEAT [31] evolves the topology,
the components, and hyperparameters simultaneously for
cheaper computation. Cai, et al. [5] use reinforcement learning
as a meta-controller and Bi-LSTM as an encoder for functionpreserving
transformations to make the network wider or
deeper. Zoph, et al. [3] use a recurrent neural network as a controller
to learn an architectural building block (cell) on a small
dataset and then transfer the block to a larger dataset. Hierarchical
genetic representation is proposed to mutate from a low
level to a high level for complex topologies [32].
Although a lot of advances have been made in NAE,
few works have investigated unsupervised learning, partially
because of the inefficiency in the search process. We design
a new search block and limit the model size according to
the initial architecture during evolution. The search efficiency
of our method can also be significantly improved
by abandoning operations with less potential in terms of
contrastive loss.
IEEE Computational Intelligence Magazine - August 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2021
