IEEE Solid-State Circuits Magazine - Fall 2017 - 60

Data reuse can be exploited by
reusing the same data across multiple parallel execution units or, equivalently, across multiple time steps on
the same execution unit. In this topology, three extreme cases can be distinguished, as shown in Figure 8.
The first multiplies the same input
data value with several weights of
a layer's different output channels.
This is also called weight parallel or
input stationary. In this implementation, every input will ideally be loaded
into the system only once. This, however, has negative repercussions on
the weight memory bandwidth, as
the weights must be reloaded frequently (every time a new input is
applied). Moreover, the accumulation of the output o cannot be performed across different clock cycles,

requiring intermediate accumulation
results o to be pushed into memory and refetched later, strongly impacting the input/output memory
bandwidth. A similar scheme fetches
every weight once and multiplies it
with many input values. This "weight
stationarity" or "input parallelism"
improves the weight memory bandwidth, yet at the expense of the input memory bandwidth. Finally, the
output stationary scheme reloads
new weights and inputs every single
clock cycle and yet is able to accumulate the intermediate results locally
within the MAC unit across different
clock cycles, to the benefit of the output memory bandwidth.
All these optimizations can be
seen as a reshuffling of the nested
loops in the pseudocode of Figure 4.
Of course, in practice, most realizations implement a hybrid form of
the three presented extreme cases.
Examples include [23] and [24], where
a two-dimensional (2-D) data path
multiplies every input with several
weights, while every weight is also
multiplied with several inputs, and
[10], where the input and output
stationarities are optimized to minimize the chip input/output bandwidth. Which parallelization scheme
is optimal depends strongly on the
network's dimensions; the parameters F, H, C, K, and M, which allow
cooptimization of the hardware; and

MAC Array
Weight Memory

Input/Output Memory

Minimize Data
Movements

×
+

FSM or
Processor
Controlled

Maximize
Parallelism

Maintain
Flexibility

FIGURE 7: Custom deep neural network processors gain efficiency by minimizing data movements and maximizing parallelism. Still, it is crucial not to lose all flexibility in mapping a
wide variety of networks. FSM: final state machine.

+
+
+
+

Inputs

×
×
×
×

Weights

Outputs

+
+
+
+

Inputs

Input

× × × ×

Weight

Outputs

Weights

+
×

Input

Providing data to all these functional units in parallel would be nearly impossible if the temporal and
spatial locality of the data was not
exploited. Indeed, many computations within one network layer share
common inputs. More specifically, as
highlighted in the pseudocode shown
in Figure 4, every weight parameter
is reused approximately M 2 times
across multiple convolutions of the
same slice in the output tensor, and
every input data point is reused
across F different slices of the output tensor. Moreover, the intermediate accumulation results o have to be
accumulated C.K 2 times. This can, in
a custom accelerator, be exploited in
several ways to further boost efficiencies beyond the highly parallel, yet
not data-flow-optimized, GPUs.

×
×
×
×

Input Stationary
(Weight Parallel)

Weight Stationary
(Input Parallel)

Output Stationary

Hybrids

Input BW

Low

High

Medium

Weight BW

High

Low

High

Medium

Output BW

High

Low

Medium

×
×
×
×

FIGURE 8: Different architectural topologies allow data reuse to be maximized, reusing either inputs, weights, intermediate results, or a
combination of the three. BW: bandwidth

FA L L 2 0 17

IEEE SOLID-STATE CIRCUITS MAGAZINE

Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Fall 2017