IEEE Circuits and Systems Magazine - Q4 2020 - 69

- onvolution structure, i.e., convolutional neural network
c
(CNN). The convolution structure embeds the raw image
into a group of two-dimensional maps. And, each neuron
of CNN directly samples a local area of the map generated
by the previous layer. This way of information processing
allows the neuron to effectively capture the local spatial
characteristics from the raw image. Fig. 3 shows the structure of standard CNN, which is called LeNet-5.
B. Details of Architecture
The whole architecture of LeNet-5 consists of two parts:
convolution module and full connection module. The
convolution module is composed of 2 convolutional layers, each of them is followed by a sub-sampling layer. In
each convolutional layer, there are lots of convolutional
units (namely convolutional filters or kernels) shifting
at a fixed stride across the input plane (such as a raw
image or a 2-D feature maps). As a simulation of those
special individual neurons in the animals' visual system, each convolutional kernel gives a strong response
when meeting a specific image pattern during the shifting and gives weaker responses to other areas. All these
responses form a feature map which reflects the spatial
distribution of the corresponding image pattern in the
input plane. In other words, the image pattern is transformed by the convolutional kernel from the original
form to a representation in a higher level. With the stack
of the kernels, more image patterns are transformed to
high-level representations and the perception of the
raw image can be obtained via combining these representations. Specifically, the convolution module mainly
contains three key ideas: local receptive field, weightsharing and spatial sub-sampling.
â â Local receptive field. Each convolutional kernel
consists of a weight matrix which has a dot product with the input elements located in a small
neighborhood (see the black squares in Fig. 3).

This structure is similar to a local receptive field
of the animal visual cortex organization. Thus,
each convolutional kernel focuses on one local
pattern of the input plane.
â â Weight sharing. During the slide of a convolutional kernel across the input plane, the weights
of the kernel are shared for each local receptive
field of the input. Obviously, much fewer parameters are required during the forward propagation due to this operation in contrast to full-connected neurons.
â â Sub-sampling. The sub-sampling is to replace elements in a local area of the feature map with
their mean value (mean-pooling) or the maximum
(max-pooling). Via such an operation, the pattern
of the local area is represented by just one element, rather than the spatial arrangement of all
elements. Moreover, due to that the number of
parameters is also reduced, the risk of overfitting
can be lowered while training the network.
The last layer of the convolution module transforms
the feature maps of the previous layer to a feature vector.
Then, the full connection module, which equals a multilayer perceptron, transforms the high-level feature vector to the discriminated result. Consequently, a mapping
from the raw image to the output is constructed via these
two modules. Formally, the whole architecture can be represented by a function as follows:

y = F ^x, " i i ,h, (1)

where x and y are respectively the input and output
of CNN. " i i , denotes the set of parameters of different
layers, which can be learned directly from data via the
error-back propagation algorithm or other advanced
learning algorithms (such as Levenberg-Marquardt and
neuron by neuron) [33].

Convolution Module

Full Connection Module

Input
Output

Convolution

Sub-Sampling

Convolution

Sub-Sampling

Gaussian
Full
Connection
Connection
Full
Connection

Figure 3. The structure of LeNet-5 consists of two parts: the convolution module and the full connection module [7].

FOURTH QUARTER 2020

IEEE CIRCUITS AND SYSTEMS MAGAZINE

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020