IEEE Circuits and Systems Magazine - Q4 2020 - 70

C. Evolution of CNNs
In addition to being motivated by Hubel's discovery [5],
CNN can also be improved based on another observation in neuroscience. There is an activation function,
commonly sigmoid function, performing the nonlinear
transformation to the outcome of each convolutional
kernel. However, nearly half neurons of the network are
activated while using sigmoid function to compute the
activation, which goes against the neuroscience observation that only 1% ~ 4% neurons in brain fire simultaneously [34]. Therefore, rectified linear unit (ReLU) is
presented as a more accurate activation model of brain
neurons that contains three characteristics: one-sided
inhibition, wide exciting boundaries and sparsity [35].
In recent years, many variants of CNN have been proposed to improve the network performance. As mentioned
in Section II-A and II-B, the receptive field of convolutional
kernels are transformed in [23] and [24], which increases
the robustness of CNN to the scale and shape transformation of objects. Parallel convolution in GoogLeNet [25] allows the network to capture features with multiple scales
and shortcut connection in ResNet [26] makes it possible
to train a network with more than 100 layers. Based on
ResNet, DenseNet [36] contains denser shortcut connections. Each layer in DenseNet takes as input the feature
maps of all the layers in front of it. That is to say, the features of a shallower layer are reused by all the layers behind, which improves the performance further.
In terms of lightweight model, some studies divide
the convolution operation into many groups to accelerate the inference through reducing the amount of
computation [37], [38]. Given an input feature with size
of 64 # 64 # 28 and a convolutional kernel with size of
9 # 9 # 16, then the amount of computation in a convolution is 28 # 9 # 9 # 16 = 36288. While with the group convolution, the input feature is divided into 2 groups firstly. Then,
each group ^64 # 64 # 14 h is filtered by half of the original
convolutional kernel ^9 # 9 # 8 h respectively. Therefore,
the amount of computation is 14 # 9 # 9 # 8 # 2 = 18144. In
addition, the parameters can be decreased by the separation of the convolution [39]. This variant separates standard convolution (3-D) to convolution inside each channel
(2-D) and 1 # 1 convolution among channels (1-D).
Besides, attention mechanism is also adopted to improve CNN's ability of capturing the key information. In
general, the attention mechanism is implemented via
multiplying a mask with corresponding attention domain.
On the one hand, the mask can be adopted in the spatial
domain to highlight the significant spatial area in feature
maps [40]. On the other hand, attention mechanism can be
applied in channel domain to focus on key feature channels
[41]. Moreover, it is also practicable to introduce attention
mechanism into both spatial and channel domain [42].
70â

IV. From Memory to RNNs
This section mainly focuses on recurrent neural network (RNN) and how it is motivated from the perspective of information transmission.
A. Motivation on Information Transmission of RNN
Even the simplest human behavior can be broken down
into a variety of serially ordered action sequences. Therefore, we need a model to process the sequential patterns
of data like speech and context. An intuitive approach is
to process temporal data spatially. However, one shortcoming of this kind of methods is that the duration of
input patterns is required to be fixed, which breaks the
completeness of temporal sequential information. Another problem is that, it is hard for such an approach to
recognize absolute displacements of patterns with similar relative structure.
For this reason, Elman proposes a network architecture
with a structure of recurrent connection that represents
time implicitly [11]. This new type of neural network is
named as recurrent neural network (RNN). The network
will make an inference for the data of each time step. In addition, the recurrent connection constructs a path from the
hidden state of the previous time step to the input of the
current time step. Therefore, RNN can reserve the " memory " about the past information when making current inference. To some extend, this process actually fits the cognition mechanism of human brains: the understanding of
an object mainly depends on our memory rather than the
information we receive currently.
B. Details of Architecture
RNN takes as input one element of the sequential data
at each time step. Meanwhile, as shown in Fig. 4(a),
there is an additional layer of context neurons that can
preserve the hidden state of the previous time step. At
each time step t, the hidden layer will give a hidden state s t
through integrating the new information from input x t and
previous information, i.e., hidden state s t - 1 (see Fig. 4(b)).
This process can be formulated as:

s t = h ^U Â· x t - 1 + W Â· s t - 1 + b h h,(2)
y t = g ^V Â·s t + b g h,(3)

where b h and b g are biases, h denotes a hyperbolic tangent (tanh) function and g denotes a softmax function.
Due to that all functions that are adopted to obtain
the actual output of RNN are differentiable, the network
architecture can be trained using backpropagation [44].
The backpropagation algorithm for RNN is essentially
equivalent to that for other network architectures. It is
important to notice that the process of updating W and
U is a little complex. For W, assuming that L t is the error

IEEE CIRCUITS AND SYSTEMS MAGAZINE

FOURTH QUARTER 2020

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020