IEEE Circuits and Systems Magazine - Q4 2020 - 67

(see Fig. 2). Nevertheless, the progress of the AI research
seems to be slowing and many advances in neural network
algorithms are actually not effective as they are expected
[22]. Therefore, it might be a good choice to look back
now, reviewing the important advances in DNNs and rethinking how those classic neural architectures are developed and why they can work. We believe that such a
research is helpful for further promoting the progress of
DNNs. In this paper, we attempt to provide an insight
of DNNs from the perspective of their intrinsic motivations. The rest of this paper is organized as follows. In
Section II, the influence domain of network motivations is
defined and coarsely categorized to help understand the
effect of different motivations. Then in Section III-V, convolutional neural networks, recurrent neural networks
and generative adversarial networks are introduced respectively to illustrate how the architectures are motivated and constructed. Evolution of the three deep neural
architectures are also discussed in these sections. In Section VI, several conclusions about intrinsic motivations of
DNNs are summarized and some promising research topics are discussed as well.
II. Influence Domain of Motivation on
Deep Neural Architectures
Since the first mathematical model of biological neuron was proposed, new neural networks have been
appearing constantly such that listing all network architectures seems to be practically impossible. Considering the diversity of neural architectures' intrinsic
motivation, we attempt to find and analyze the general
characteristics of motivations in DNNs. For clarity, we
introduce the concept of influence domain to represent
which part of a neural architecture is inspired and motivated. Through reviewing the proposal and evolution of
some deep neural architectures, we roughly divide the
influence domain of different network motivations into
three categories: information processing, information
transmission and learning strategy.
A. Information Processing
Artificial neuron is the basic information processing unit
in neural networks. How neurons work and how information is processed in the network directly affect the network's ability of representation learning. Hence, many
deep neural architectures are constructed with their
own way of information processing. Motivated by the
study of visual cortex of cat, a convolutional structure
is presented in neocognitron and convolutional neural
network (CNN). The convolutional structure consists of
neurons arranged in columns, which is termed as convolutional kernel. Such an aggregation of neurons enables
neural network to learn representations directly from
FOURTH QUARTER 2020

Complex Networks

Video Security

Bioinformatics

Electronic Commerce

Internet of Things

Intelligent Manufacturing

Figure 2. Present and potential application domains of deep
neural architectures.

the raw image, avoiding plenty of feature engineering
in conventional machine learning. On this basis, some
improved convolutional structures are proposed to enhance the network performance. Dilated convolution
[23] allows the receptive field of convolutional kernel
to be enlarged, thus can capture more information from
larger spatial region. Deformable convolution [24] can
learn an offset for each sampling point of convolutional
kernel, so the sampling points can be arranged irregularly. This structure leads to stronger robustness to the
scale and shape transformation of objects in image.
Despite the power of CNN and its variants in feature extraction, they can only capture the existence of patterns.
In fact, it is hard for the convolution structure to learn
the interrelationship between different patterns [15]. In
recent years, it is argued that there are a large number
of micro-columns neuron modules in cerebral cortex that
can handle different types of visual stimulus. Motivated
by this, a capsule structure is invented to solve the problem of CNNs [15]. In contrast to the convolutional kernel
that outputs a scalar to measure the existence of a pattern in current location, the capsule produces a vector
to represent the attributes of a pattern (such as the location, posture and so on). Therefore, capsule network is
more promising in tasks that have higher requirements
on feature representation, such as image segmentation
and object detection. Moreover, due to the abundant information in the output vector of capsule, neural network
is endowed with better interpretability.
B. Information Transmission
Another important aspect of neural architectures is how
information flows in the network. Generally, the information transmission path can decide how hidden features are used by neurons in different layers. In comparison of standard CNN, GoogLeNet [25] contains multiple
parallel paths between two layers. In each path, features
from the previous layer are filtered by convolutional kernels with a specific scale. After that, the outputs of the
IEEE CIRCUITS AND SYSTEMS MAGAZINE

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020