IEEE Circuits and Systems Magazine - Q4 2020 - 74

objective function for network training. Meanwhile, the
realization of such a learning strategy generally needs a
matching network framework, such as the encoder-decoder framework for self-supervised learning and GAN
framework for adversarial learning.
In order to grasp the future trend, some promising research topics about deep neural architectures are also
discussed in this paper.
â â Neural networks based on brain mechanisms: The
architecture of DNNs can be regarded as an imitation of the hierarchical structure of human speech
perception and production systems [62]. In the case
of convolutional neural networks, the raw image information is transformed into a more abstract representation layer by layer until final cognition is formed.
With the deepening understanding of the brain nerve
structure and information processing mechanism,
novel neural networks based on brain mechanisms
are showing a level of intelligence closer to that of
human. At present, a typical brain-like neural network is spiking neural network (SNN) [63], [64]. The
model of SNN is constructed to be closer to the working mechanism of biological neural networks. Unlike
traditional neural networks, the information carrier
processed by neurons in SNN is impulse train. This
way of information processing is in fact a simulation
of the accumulation of membrane potential and impulse discharge of biological neurons after reaching
the threshold. These bionic characteristics lead to
stronger capability of complex information processing. However, it is hard to train a SNN due to the nondifferentiability of neurons. Thus, a general learning
strategy for such a novel neural architecture is a significant research direction.
â â Automatically designing network architecture:
Deep neural networks have attained outperforming achievement in the automation of feature extraction, but they still suffer from the high price
to acquire a good network architecture. Designing
an appropriate network aiming at specified task
generally requires a large amount of time prior expertise for trial and exploration. Thus, how to efficiently search an available network architecture
in low cost is destined to be a research hotspot
in the future. Neural Architecture Search (NAS) is
an area of greatest interest in automating machine
learning. This technique is used to automatically
design network architecture by exploring network structures and hyperparameters in specified search space with some strategies. Elsken
et al. [65] summarize existed researches in this
field and analyze these works from three aspects:
search space, search strategy and performance es74â

timation strategy. At present, most studies of NAS
are conducted for image classification. More architectures for other domains, such as image restoration, semantic segmentation and natural language
processing, are waiting to be further explored.
â â Training neural network via contrastive learning: GAN is essentially a generative model that
needs to learn to construct as more sample details as possible. But sometimes, the network
only needs to learn a representation which
can used to distinguish different samples. The
learning strategy motivated by this idea is contrastive learning. For a particular sample x, we
can construct some positive samples " x + , and
negative samples " x - , through a certain transformation. The objective of contrastive learning
is to maximize the consistency between f(x) and
f ^ x +h while minimize that between f(x) and f ^ x -h,
where f is a mapping function. Related mechanism and framework of contrastive learning can
be seen in [66], [67]. About contrastive learning,
there are two key questions to be answered. The
first one is how to define a metric to measure the
difference between the representation of samples. An effective metric is indispensable to an
appropriate objective function for contrastive
learning. The second one is how to construct the
transformation to generate appropriate positive
and negative samples. For data from different domains, such as image, text and audio signal, it is
important to find corresponding transformation
that can highlight the semantic relation between
sample pair.
Acknowledgments
This work was supported in part by the National Key
Research and Development Program of China under
Grants 2018AAA0101100, 2016YFB0800401, in part by
the National Natural Science Foundation of China under Grants 61621003, 61903017, 61532020, and in part by
the China Postdoctoral Science Foundation under Grant
2020M670087.
Weilin Luo received his B.S. degree in information and computing science and
M.S. degree in automation science from
Nanjing University of Aeronautics and Astronautics, Nanjing, China in 2018. He is
currently pursuing the Ph.D. degree with
the school of Automation science and Electrical Engineering, Beihang University, Beijing, China. His current research interests include intellisense and intelligent decision
making techniques based on deep neural architectures.

IEEE CIRCUITS AND SYSTEMS MAGAZINE

FOURTH QUARTER 2020

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020