IEEE Circuits and Systems Magazine - Q4 2020 - 68

- onvolutional kernels from different paths are concatc
enated together as the input of the next layer. Consequently, features with different scales can be captured
through this structure, which improves the network's
capability of object detection. In addition, ResNet [26] is
proposed that adds shortcut connections from shallower layer to deeper layer. This shortcut connection is able
to ease the training of network with very deep architecture, because it actually equals to an identity mapping
which allows the gradient to be propagated to shallower
layers without vanishing during the backpropagation.
The information can not only flow forward during
the network inference, but also be transmitted in more
diverse ways in some neural architectures. These special ways of information transmission lead to an ability
of handling data with different forms. To address temporal sequential data, RNN takes as input the data at each
time step respectively. The recurrent connection makes
hidden state of the previous time step flow to the input
of current time step. Thus, RNN can learn a representation from the temporal sequential data, aggregating both
the information of the current and that of the past. Furthermore, the recurrent connection can be improved
by adding some " gates " to enhance, suppress or even
discard the information flow from the previous time step
[27]. Another instance is Graph neural network (GNN).
This network architecture is able to deal with graph data.
Through the edges in the input graph, the information of
each node will be aggregated with that of adjacent nodes.
Being fed this combination of node information, the neural network can learn a graph embedding in accordance
with the topology of graph [28] and realize the classification of each node or the whole graph. With larger field of
information aggregation, the information can even flow
from farther away nodes to the central node.
C. Learning Strategy
Besides the processing and transmission of information,
how a deep neural architecture learns from data is also
of great significance. The learning strategy mainly determines what the network can learn and what kind of
tasks it can fulfill. For most deep neural architectures,
the network parameters can be trained end-to-end
through supervised learning technique. It allows the
network to learn a mapping from the input data to a desired output. In other words, the network learns a conditional probability density function p ^ y x h, where x and
y denote the data and desired output respectively. On
this basis, the strong fitting ability of deep neural architectures makes them popular for many discriminative
tasks such as recognition, detection and classification.
Moreover, deep neural architectures can directly learn
the distribution of data itself, i.e., p(x). Autoencoder [9]
68â

is designed as an encoder-decoder framework which can
be used to compress data. The encoder part transforms
the data to a low-dimensional representation while the decoder restores the original input from the representation.
For this purpose, the entire architecture is trained through
a self-supervised learning strategy: minimizing the reconstruction error between the restored data and the raw
data. Another famous leaning strategy in recent years is
adversarial learning. It is adopted to train the architecture
of GAN [13] which contains two networks: generator and
discriminator. The objective of adversarial learning is to
enable the generator to learn the data distribution implicity through a zero-sum game between the two networks.
According to the three influence domains of intrinsic
motivations, we can further discuss how DNNs are motivated and developed. In the following three sections,
CNN, RNN and GAN are taken as instances respectively
to illustrate the influence of their own motivations to the
architecture. Specifically in each section, the intrinsic
motivation of a DNN and the influence domain of motivation are discussed firstly in each section. Then, details of
the neural architecture are also demonstrated to deepen
the understanding of the network structure. At the end
of each of next three sections, some variants of the DNN
are introduced to provide more knowledge about corresponding neural architectures.
III. From Animal Visual Cortex to CNNs
This section mainly focuses on convolutional neural
network (CNN) and how it is motivated from the perspective of information processing.
A. Motivation on Information Processing of CNN
CNN is one of the deep discriminative models, which
has been proven to be pretty effective in many computer vision tasks such as object detection [29], object
tracking [30] and video classification [31]. The birth
of CNN integrates computer science, mathematics and
most importantly, neurobiology. In an interesting experiment, Hubel and Wiesel [5] found that there are a
small number of cells in the visual cortex that are sensitive to specific vision areas. They discovered that some
individual neurons in animals visual system can transmit electrical signal in response to certain properties of
visual sensory inputs, such as the edges with a specific
orientation [32].
In traditional neural networks, the hidden state is represented as a one-dimensional vector and each neuron
takes as input the entire state from the previous layer.
For image data, embedding a raw image into an onedimensional representation actually destroys the spatial
characteristics of the data. Inspired by Hubel's study,
Lecun et al. [7] designed a neural architecture based on

IEEE CIRCUITS AND SYSTEMS MAGAZINE

FOURTH QUARTER 2020

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020