IEEE Consumer Electronics Magazine - April 2017 - 51

simplest way is the difference between the predicted (learned)
and the true outputs.
Training algorithms work by updating the weights and measuring the way that the loss changes over time. This is usually
accomplished by some form of a gradient descent optimizer. It's
possible to become stuck or quickly converge to a nonoptimal
solution when strictly following the steepest gradients. For this
reason, we typically use some form of stochastic gradient descent, which is simply a stochastic or slightly randomized form
of gradient descent. Back propagation is used to allow us to
propagate the error information from the last layer to the first
layer to modify the weights, and this is often used in a way that
is synonymous with training. Methods that can be used to improve a network's results are called learning rules.

a renaIssance for neural networks?
Although these methods have been successfully used for decades, they have seen a very recent resurgence as refinements
upon existing techniques, combined with newer hardware and
the growth of big data, have created an artificial intelligence
boom that shows no signs of slowing down. The global market
for smart machines is expected to exceed 15 billion by 2019,
with an average annual growth rate of nearly 20% [8]. The set
of techniques that have led to this growth has been coined deep
learning. In the following sections, we will discuss some of the
techniques that enable deep learning, why they work, and how
they are used to make smart machines.

Convolutional neural networks
To store and process analog signals (e.g., voice and biological
signals) on a computer, one must first convert these inputs
into a digital form in a process called analog-to-digital conversion. This transforms the information from a continuous
space to a discrete space. Some information is lost in the process. Often that information isn't critical to understanding the
underlying analog signal, but in some instances, we could
lose critical data, such as high-frequency information.
In digital processing, we often refer to a piece of data, such
as a picture, as a sample. It is convenient, but computationally
expensive, to represent these samples in a high-dimensional space
where each unit (or pixel, in the case of images), is considered as
being located on a specific axis with the range of possible values
being the size of that axis (e.g., 0-255 in the case of an 8-b color-channel in an image). An image that is 100 × 100 pixels would
be represented as a vector that is a point in a 10,000-dimensional
space. We call this space the feature space. Since even the most
powerful computers can have trouble with such high-dimensional space, we try to reduce the number of dimensions to just those
that are critical to a task or to change the way these features are
represented. In the traditional pattern recognition approach, to
perform a given task, we separate our process into two steps:
feature generation and feature selection. The former generates
new features from the pixel space, while the latter reduces the
dimensionality of the feature space. Examples of feature generation include morphological, Fourier, and wavelet transforms,
which create more useful features for specific tasks. Feature se-

lection includes methods like principal component analysis and
linear Fisher discriminant [9].
A newer technique based on sparse mapping calls for
expanding the dimensions with the goal of representing more
abstract features instead of trying to reduce the dimensions.
This is inspired by models of the visual cortex of animals
[10]. Convolutional layers, a key component of deep learning, make use of this sparse mapping approach.

convolutIon and Its role In a neural network
One of the most novel and useful aspects of CNNs is that
they can learn the filters that previously had to be customdesigned by the researcher, a task that would often take years
of trial and error. Convolutional layers are essential to this
task in modern deep neural networks.
Convolutional layers make use of the convolution operator.
The convolution operator is used on two functions. One is the
signal from the sample space and the other, called the filter, is
applied to the sample. On a GPU, convolutions are implemented as matrix multiplications. This operator has a long history in
image processing applications and dates back to the time when
digital image processing started. The convolution operation
can be discussed in both spatial and transform space. In the
spatial space, the convolution operator is the equivalent operation of correlation with the reversed filter, i.e., this operator
calculates the similarity of the input function with the filter.
For example, the edge detection and corner detection filters use
the similarity of the input image with a predefined filter mimicking the edge or corner shape. Looking at the function in the
transform space, the convolution is performing frequency filtering. For example, low-pass or high-pass filters have their
equivalent spatial filters, which could be applied to the image
using convolution operations (see Figure 4).
In the deep learning approach, the filter is learned and
applied to the data during the training process with the hope
that, after training, the learned filter will be the best choice
for the task. One difference between the way convolutions are
used in CNNs from more traditional uses is that the convolution operator is applied using a four-dimensional (4-D) filter.
This is essentially a set of three-dimensional (3-D) filters that
are stacked in the fourth dimension. We use 4-D filter to map
one 3-D space to another 3-D space (see Figure 5).

ConvoLutionaL Layer
In general, for an n-dimensional signal, the convolutional
layer is an n- or (n + 1)-dimensional feature space mapping
with n + 1 or n + 2 dimensional kernels (filters). For example, given a two-dimensional image, the convolutional layer
would be 3-D with a 4-D kernel. In this case, the four dimensions of the kernel correspond to
▼▼ the width of the input
▼▼ the height of the input
▼▼ the number of channels of the input
▼▼ the number of channels of the output.
In Figure 5, you can see two different convolutional layers. The
k channel layer on the left side is mapped to a p channel layer on
APRIL 2017

IEEE Consumer Electronics Magazine

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - April 2017