IEEE Circuits and Systems Magazine - Q4 2019 - 25

(2)

where 0 ij is a local neighborhood around location ^i, j h
in the kth feature map of the lth layer [53]. Note that the
weights and the bias are shared by the neurons belonging
to the same layer. From (1) and (2) it is obvious that the
changes between the classical NN architectures and the
CNNs are only in the layer category and configuration.
High-level feature representations can be extracted
gradually by stacking several convolutional and pooling
layers. The one or more fully-connected layers added
before the output layer aim to learn nonlinear combinations of the high-level features passed from the previous layers and convey the higher-level representation
to output layers. Fully-connected layers can be optionally replaced by convolution layers, of which the size of
the convolution kernels is 1 # 1 [54]. The last layer in
a CNN is known as the output layer, which is configured
based on the type of task that the network needs to

F6:
Layer
84

Full
Connection

Subsampling

Convolutions

C5:
P1: Feature Maps C3: Feature Maps
P2:
Layer
16
at
10
×
10
6 at 14 × 14
Feature Maps 120
16 at 5 × 5

Subsampling

Convolutions

Input
32 × 32

C1: Feature Maps
6 at 28 × 28

y li, j, k = pool ^f lm, n, k ^w lk x lm, n + b lkhh, 6 ^m, n h ! 0 ij

Output 10

Gaussian
Connections

2) Convolutional Neural Networks
Convolutional Neural Networks (CNNs) have been usually exploited in the fields of image classification, object detection, face recognition and speech processing.
Compared with general DNNs, matrix multiplication is
replaced by the convolution to dramatically reduce the
amount of weights in modeling. This makes CNNs a powerful tool to relieve the memory curse in image processing related tasks since the large number of pixels in a
single picture always leads to a similarly large number
network parameters. Consequently, the complexity of
the network can be decreased. Another advantage of
CNNs is that the images, as raw inputs, can be directly
imported to the network without the feature extraction
procedure. In addition, with a workstation equipped
with GPUs, training a CNN is more efficient.
Generally, a basic CNN includes three types of layers:
convolutional, pooling and fully-connected layers. For
example, Fig. 6 demonstrates the famous LeNet-5. In this
network, there are three convolutional layers, two pooling layers and two fully-connected layers. For each convolutional layer, there are several convolution kernels
to compute feature maps from the previous layer. Distinct feature representations of original inputs can be
learned by the convolution procedure and transferred
to subsequent layers. A region of neighboring neurons
are mapped to single neuron on the subsequent layer
and this region is referred to as the receptive field of
this single neuron. To achieve a feature map, the input
is convolved by a trainable kernel and the result is ap-

plied with an element-wise nonlinear activation function. Due to the property of parameter sharing in CNNs,
the kernels are shared by all pixel areas of the input.
Finally, all the feature maps are obtained by utilizing a
variety of convolutional kernels. The pooling layers are
introduced to reduce the overall size of the signal to
avoid over-fitting problems caused by high dimensional
inputs. As in classical neural networks like the MLP, neurons in all types of layers compute the dot product between the input vectors and the weights, to which biases
are added. Then the weighted sum is passed through a
nonlinear activation function to obtain the output vectors. Denoting the pooling function as pool ^ · h , for each
feature map we have:

Full
Connection

predict the output in practical pattern recognition problems. More details of the backpropagation algorithm
can be found in [52].

Figure 6. Architecture of LeNet-5 [29].

FOURTH QUARTER 2019

IEEE CIRCUITS AND SYSTEMS MAGAZINE

IEEE Circuits and Systems Magazine - Q4 2019

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2019