IEEE Solid-State Circuits Magazine - Fall 2017 - 56

typically trained through supervised
learning, whereby a machine learns
a generalized model from many
training examples, enabling it to
classify new items.

i11
i12
i13

The trained classification model
in such neural networks consists of
several layers of neurons, wherein
each neuron of one layer connects
to each neuron of the next layer, as

o11

o21

o31

Car?

o12

o22

o32

House?

i14

o ln = v c /w lmn . i lm + b ln m .

(1)

i15

o11

o33

o23

o13

i11
i12
i13

i11

× w111

i12

× w112

i13

× w113

Dog?

b11
o11

+
σ

Oln = σ (∑wlmn × ilm + bln)
m

FIGURE 1: A traditional fully connected neural network is made up of layers of neurons.
Every neuron makes a weighted sum of all its inputs, followed by a nonlinear transformation.

Edges
Gradients
Corners
HOG
...
Image

Designed
Feature
Extraction

Neural
Network

"House"

Trained
Classifier

Class
Label

Neural
network

"House"

(a)
...
...
...

Image

Trained
Feature
Extraction

Trained
Classifier

Class
Label

(b)

"House"

Image

Trained
Feature
Extraction

Trained
Classifier

Class
Label

(c)
FIGURE 2: (a) Traditionally, machine learning classifiers were trained and applied on handcrafted features. (b) The advent of deep learning allowed the network to learn and extract the
optimal feature sets. (c) Such a network trains itself to extract very coarse, low-level features
in its first layers, then finer, higher-level features in its intermediate layers, and, finally, targets
full objects in the last layers. HOG: histogram of oriented gradients.

illustrated in Figure 1. The output
of the network indicates the probability that a certain object class is
observed at the network's input. In
such a network, every individual neuron creates one output o, which is a
weighted sum of its inputs i. For the
nth neuron, of layer l, this can be formalized as

FA L L 2 0 17

IEEE SOLID-STATE CIRCUITS MAGAZINE

The weights w lmn and biases b ln are
the flexible parameters of the network that enable it to represent a
particular desired input/output mapping for the targeted classification.
They are trained with supervised
training examples in an initial offline training phase, after which the
network can classify new examples
presented to its inputs, a process typically referred to as inference.
Such neural networks have been
used for decades in several application domains. In a classical patternrecognition pipeline [Figure 2(a)],
features are generated from an input
image by an application-specific feature extractor, hand-designed by an
expert engineer. This preliminary
feature extraction step was necessary
because, at that time, one could use
only small neural networks with a
limited number of layers that did not
have the modeling capacity required
for complex feature extraction from
raw data. Larger neural networks were
impossible to train due to nonconvergence issues, lack of sufficiently
large data sets, and insufficient compute power.
Yet, after a long winter for neural
networks in the 1970s and 1980s,
they regained momentum in the
1990s and again in the 2010s. The
incr e a sing av a ilabilit y of powerful compute servers and graphics processing units (GPUs), the
abundance of digital data sources,
and innovations in training mechanisms allowed training deeper and
deeper networks, with many layers of
neurons. This meant the start of a new
era for classification, as it allow ed training networks with enough

Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Fall 2017