IEEE Solid-State Circuits Magazine - Fall 2017 - 57

Deep Neural Network Topologies
Another crucial factor in the breakthrough of deep learning technology is the advent of new network
topologies. Classical neural networks-
which rely on so-called fully connected layers, with each neuron of
one layer connected to each neuron
of the next layer (Figure 1) -suffer
from a very large number of training
parameters. For a network with L
layers of N neurons each, L. (N 2 + N)
parameters must be trained. Knowing that N can easily reach the
order of a million (e.g., for images
with a million pixels), this large
parameter set becomes unpractical
and untrainable.
For many tasks (mainly in image
processing and computer vision),
convolutional neural networks (CNNs)
are more efficient. These CNNs, in spired by visual neuroscience, organize the data in every network layer
as three-dimensional (3-D) tensors.

To enable efficient evaluation of deep
neural networks, optimizations at both the
algorithmic and hardware level are required.
size H # H # C ) into a 3-D output tensor I (of size M # M # F ).
As illustrated in Figure 4, each
element of the output tensor O does
not need all elements of the input
tensor I to be computed. Instead, it

The first part of the network consists out of a sequence of convolutional layers and pooling layers,
replacing the traditional fully connected layers. A convolutional layer
transforms a 3-D input tensor O (of

Hand-Crafted
Features
28.2
25.8

Deep
Learning

ImageNet Challenge:
1,000 Classes
1.3 M Training Images/50 k
Validation/100-k Testing
Top 5 Classification Errors (%)

Eight Layers

16.4 Eight Layers
11.7

19 Layers
7.7

Human
5.1%

22 Layers
6.7

152 Layers

1

SV
Al RC
ex '1
N 2
et
IL
SV
R
C
'1
3
IL
SV
R
C
VG '14
G
IL
S
G V
oo R
gl C'1
eN 4
et
IL
SV
R RC
es '1
N 5
et

IL

'1
C
R
SV
IL

SV

R

C

'1

0

3.57

IL

modeling capacity to operate directly
on raw data. [Figure 2(b)]. Such "deep
learning networks" thus fulfilled
the role of both feature extractor
and classifier.
A deeper network can automatically learn the best possible features
during its training phase, instead of
relying on features hand-crafted by
humans. When inspecting trained
networks, one can see that a deep
neural network trains itself to extract
very coarse, low-level features in its
first layers and finer, higher-level
features in its intermediate layers
and then targets full objects in the
last layers [Figure 2(c)].
A network's ability to learn the
most optimal features significantly
boosted the classification accuracy
of such networks, resulting in their
true breakthrough: deep learning was
born. Over the last decade, deep learning has, as such, been able to move to
deeper and deeper network architectures, enabling tremendous improvements in achievable classification
accuracy, as illustrated by the results
from the yearly ImageNet challenge
(Figure 3) [2].

FIGURE 3: The classification results of the ImageNet challenge have seen enormous boosts
in accuracy since the appearance of deep learning submissions. (Data from [2].) ILSVRC: ImageNet Large-Scale Visual Recognition Challenge; AlexNet: a CNN named for Alex Krizhevsky;
VGG: a network from the Visual Geometry Group at Oxford University; ResNet: Residual Net.

Classification

Trained Feature Extraction
C

F

K
K

H

ReLU
Convolutional
Max-Pooling

M

Fully Connected
ReLU
Convolutional Max-Pooling Classification

Per Output Pixel of a Layer:
for (int f = 0; f < F; f++)
* Load C.K 2 Weights
for (int mx = 0; mx < M; my++)
* Load C.K 2 Inputs
for (int my = 0; my < M; mx++)
for (int c = 0; c < C; c++)
* Do C.K 2 MACs
for (int kx = 0; kx < K; kx++)
* One Output Store
for (int ky = 0; ky < K; ky++)
Repeat F.M 2 Times Per Layer
o [c ][mx][my] += w [f ][c ][kx][ky] . i [c ][mx + kx][my + ky]);
FIGURE 4: The topology and pseudocode of one layer of a typical CNN. The psuedocode is
for one layer of the network. MACs: multiply accumulation.

IEEE SOLID-STATE CIRCUITS MAGAZINE

FA L L 2 0 17

57
Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Fall 2017
