IEEE Circuits and Systems Magazine - Q4 2020 - 66

f-unction of biological neurons. These artificial neurons
transmit nonlinear activation through some weighted connections. Then, the network can approximate a specific
complicated function via stacking enough such transmission. Despite the significant achievements of ANNs, they
are limited to their capacity of tackling data in the raw
form [3]. Sophisticated feature engineering requiring considerable expertise is necessary for the network to learn
reliable patterns from the raw data. Inspired by the investigations to hierarchical structures of human speech system, some novel network architectures with multiple layers are developed, namely deep neural networks (DNNs).
These hierarchical neural architectures are expected to
overcome the limitation of conventional ANNs.
A historical timeline is depicted to help overview the
appearance of various influential DNN architectures. As
shown in Fig. 1, group method of data handling (GMDH)
based network was proposed in 1965 [4]. To the best of
our knowledge, it was the first time that multilayer representation learning is introduced in ANNs. Following
this sort of hierarchical structure and being inspired by
the characteristics found in the visual cortex of cats [5],
Fukushima [6] presented a genuine deep neural architecture in 1979, namely neocognitron. This model contains a
convolutional structure which has a receptive field with
given weights, can capture features from a 2-dimensional
input plane. Although the architecture is similar to convolutional neural networks (CNNs) [7], the weights of neocognitron are not trained by backpropagation-based supervised learning. Instead, the weights are given through
a winner-take-all-based unsupervised learning strategy
[8]. After that, in 1988, autoencoder [9] was developed
which can learn a low-dimensional representation of the
input data. This network firstly used a self-associated
pre-training techniques based on unsupervised learning
with an encoder-decoder framework [10], which is one

of the widely applied network frameworks today. In 1990,
recurrent neural network (RNN) [11] containing the structure of recurrent connection was designed to model the
short-term memory for neural networks. This innovation
enables RNN to process temporal sequential data effectively. Subsequently, in 1998, convolutional neural network (CNN) [7] was proposed and it achieved impressive
performance in the task of handwriting character recognition. With the structure of neocognitron, CNN adopts
backpropagation based on gradient descent algorithm to
optimize the network weights end to end.
Despite the appearance of early DNNs, the concept of
deep learning was not popular until 2006 when Hinton
et al. [12] proposed deep belief network (DBN). They exploited an unsupervised training algorithm to optimize
DBN layer by layer. This technique allows neural architectures with more layers to be trained efficiently and effectively, thus attracts lots of attention to deep learning.
In 2014, the framework of generative adversarial network
(GAN) [13] was presented being inspired by the game
theory. As a generative model with a dualistic framework
[14], the two sub-networks in GAN can be trained through
the game between the two networks and can learn data
distribution implicitly. After that in 2017, a novel capsule
structure was proposed in capsule network (CN) [15].
This structure is a special artificial neuron in which vectors rather than scalars are taken as the information carrier. Such a characteristic enables CN to capture the spatial relation between image patterns, which can hardly be
implemented by CNNs.
These deep neural architectures, nowadays, have
turned out to be very effective in many scenarios such
as video security [16], electronic commerce [17] and intelligent manufacturing [18]. Besides, DNNs also show
great potential for the research on complex network [19],
bioinformatics [20] and internet of things [21] etc.

Covolutional Neural
Network
GMDH Based
Network
......

1979

1995

2017

2006
1998

1988

1965
Coventional
Neural Networks

Generative Adversarial
Network

Autoencoder

......

2014
Deep Belief
Network

Neocognitron
Recurrent
Neural Network

Capsule Network

Figure 1. The timeline of DNNs.

66â

IEEE CIRCUITS AND SYSTEMS MAGAZINE

FOURTH QUARTER 2020

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020