IEEE Solid-State Circuits Magazine - Fall 2017 - 59

inference in energy-scarce milliwatt or
microwatt devices. Currently, microcontrollers and embedded GPUs
are limited to efficiencies of a few
tens to hundreds of GOP/W, while
embedded inference will only be
fully enabled with efficiencies well
beyond 1 TOP/W. Overcoming this
bottleneck is possible yet requires
a tight interplay between algorithmic optimization (modifying the
network topology) and hardware
optimization (modifying the processing architectures).
The following section elaborates on
the most promising optimizations currently being explored toward energy-efficient, embedded deep inference. The
focus here is on the energy-efficient
execution of convolutional layers,
which form the bulk of the workload
during inference. However, several techniques can also be applied to fully connected layers.

Algorithmic and Architectural
Techniques for Energy Efficiency
GPUs and central processing units
(CPUs) are extremely flexible, generalpurpose machines. While this makes
them widely deployable and easy to use
and program, it also limits their efficiency because they cannot exploit
several computational aspects of
deep inference networks, resulting
in both a memory bottleneck and a
computational bottleneck. More specifically, deep inference networks
have three typical characteristics
that can be exploited-or further
enhanced-to improve execution energy efficiency:
1) Deep learning networks exhibit a
very particular data flow with a
large amount of potential parallelism and data reuse. This can,
moreover, be manipulated during network training by playing
with the F, H, C, K, and M parameters of the network.
2) Deep learning networks prove
to be quite robust to approximations or fault introductions. This
is exploited in various reducedprecision hardware implementations. Also, this characteristic can

CNNs, inspired by visual neuroscience,
organize the data in every network layer
as 3-D tensors.
be manipulated when training the
network, allowing it to find the
best tradeoff between a low complexity and a robust network.
3) Deep learning networks demonstrate large sparsity. Many
parameters become very small,
even equal to zero, after network
training. Also, many data values
propagated with the network
during evaluation become zero.
This can be exploited to reduce
operations and memory fetches
in hardware yet can also be stimulated further with innovative
training techniques.
We will show how, for each of these
three aspects, hardware can benefit
from the network's characteristics
but also how, during the algorithmic training phase of the network,
it is possible to additionally optimize the particular characteristic to
reach even greater efficiency gains.
As such, it is clear that the hardware
and algorithmic level need to closely
cooperate not only to exploit but also
to enhance the network's characteristics toward the most efficient hardware-software realization. All of the

techniques highlighted in this article
are summarized in Figure 6.

Enhancing and Exploiting
Network Structure
In many application areas, designers
have improved the energy efficiency
of embedded network evaluation by
moving away from general-purpose
processors and developing customized hardware accelerators. Such
accelerators can exploit the known
data flows within the algorithm to
1) enhance the parallel execution of
the algorithm as well as 2) minimize
the number of data movements (Figure 7). Descriptions of several application-specific integrated circuits
targeting the efficient execution of
convolutional and fully connected
layers have recently been published.
All solutions exhibit a very large
degree of parallelization, far beyond
CPU parallelism. This easily demonstrates itself in a data path containing a few hundred to thousands of
multiply accumulators (MACs), with
Google's recent tensor processing
unit as an extreme example (64,000
MACs) [9].

Solving the Memory Bottleneck
Algorithmic
Techniques

Tightly
Linked

Processor
Architecture
Techniques

Solving Computational Bottleneck

A) Enhancing and Exploiting Network Structure
* Spatial Data Reuse
* Highly Parallel Architectures
* Hierarchical Memory
* Distributed Processing
Exploiting Data Locality
B) Enhancing and Exploiting Fault Tolerance
* Quantized Training
* (Dynamic) Fixed Point
* Stochastic Memories
* Analog and Statistical
Processing
C) Enhancing and Exploiting Network Sparsity
* Memory and Computational
* Network Pruning
Gating
* Network Compression
* Compressed Computing
and Weight Sharing

FIGURE 6: An overview of the algorithmic and processor architecture techniques discussed to
increase efficiency and enable the inference of deep neural networks in embedded devices.

IEEE SOLID-STATE CIRCUITS MAGAZINE

FA L L 2 0 17

Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Fall 2017