IEEE Solid-States Circuits Magazine - Summer 2020 - 39

hardware is a net benefit across multiple workloads.
Flexibility has become increasingly important when we factor in
the many techniques being applied
to the DNN models with the promise
to make them more efficient, since
they increase the diversity of workloads that need to be supported.
These techniques include DNNs with
different network architectures (i.e.,
different layer shapes, which impact
the amount of required storage and
compute and the available data reuse
that can be exploited), different levels of precision (i.e., different numbers of bits across layers and data
types), and different degrees of sparsity (i.e., the number of zeros in the
data). There are also different types
of DNN layers and computations beyond MAC operations (e.g., activation
functions) that need to be supported.
Actually getting a performance or
efficiency benefit from these techniques invariably requires additional
hardware. Again, it is important that
the overhead of the additional hardware does not exceed the benefits
of these techniques. This encourages a hardware and DNN model codesign approach.
To date, exploiting the flexibility
of DNN hardware has relied on mapping processes that act like static
per-layer compilers. As the field moves
to DNN models that change dynamically, mapping processes will need
to dynamically adapt at runtime to
changes in the DNN model or input
data while still maximally exploiting
the flexibility of the hardware to improve efficiency.
In summary, to assess the flexibility of DNN processors, their efficiency (e.g., inferences per second,
inferences per joule) should be evaluated on a wide range of DNN models.
The MLPerf benchmarking workloads
are a good start; however, additional
workloads may be needed to represent efficient techniques, such as efficient network architectures, as well
as reduced precision and sparsity.
The workloads should match the desired application. Ideally, since there

can be many possible combinations,
it would also be beneficial to define
the range and limits of DNN models
that can be efficiently supported on
a given platform (e.g., the maximum
number of weights per filter or DNN
model; minimum amount of sparsity;
required structure of the sparsity;
levels of precision, such as 8, 4, 2,
or 1 b; types of layers and activation
functions; and so on).

Scalability
Scalability has become increasingly
important due to the wide use cases
for DNNs. This is demonstrated by
emerging technologies employed
not just for scaling up the size of the
chip but also for building systems
with multiple chips (often referred
to as chiplets) [32] or even waferscale chips [33]. Scalability refers to
how well a design can be scaled up
to ach ieve h igher per for m a nce
(i.e., latency and throughput) and
energy efficiency when increasing
the amount of resources (e.g., the
number of PEs and on-chip storage).
This evaluation is done under the
assumption that the system does
not have to be significantly redesigned (e.g., the design only needs
to be replicated), since major design
changes can be expensive in terms
of time and cost. Ideally, a scalable
design can be used for low-cost embedded devices and high-performance devices in the cloud simply
by scaling up the resources.
Ideally, the performance would
scale linearly and proportionally with
the number of PEs. When the problem size (e.g., the batch size) is held
constant, this is referred to as strong
scaling and is the more challenging
type of scaling. On the other hand,
scaling performance while allowing
the problem size to increase (e.g., by
increasing batch size) is called weak
scaling and is also an important objective in some situations.
Similarly, the energy efficiency
would also improve with more onchip storage; however, this would
likely be nonlinear (e.g., increasing
the on-chip storage such that the en-

tire DNN model fits on chip would
result in an abrupt improvement in
energy efficiency). In practice, this is
often challenging due to factors, such
as the reduced utilization of PEs and
the increased cost of data movement
due to long-distance interconnects.
Scalability can be connected with
cost efficiency by considering how
inferences per second per cost and
inferences per joule per cost change
with scale. For instance, if throughput increases linearly with the number of PEs (with proportional scaling
of all storage), then the inferences per
second per cost could be constant. It
is also possible for the inferences per
second per cost to improve superlinearly with an increasing number of
PEs due to increased sharing of data
across PEs. On the other hand, inferences per joule per cost might remain constant or even improve as a
consequence of more sharing of data
by multiple PEs.
In summary, to understand the
scalability of a DNN processor design, it is important to report its performance and efficiency metrics as
the number of PEs and storage capacity increase. This may include how
well the design might handle technologies used for scaling up, such as
interchip interconnect.

Interplay Among
Different Metrics
It is important that all metrics be
accounted for to fairly evaluate the
design tradeoffs. For instance, without the accuracy given for a specific
data set and task, one could run a
simple DNN model and easily claim
low power, high throughput, and low
cost-however, the processor might
not be usable for a meaningful task.
Alternatively, without reporting
the off-chip BW, one could build a
processor with only MACs and easily claim low cost, high throughput,
high accuracy, and low chip power-
however, when evaluating system
power, the off-chip memory access
would be substantial. Finally, the
test setup should also be reported,
including whether the results are

IEEE SOLID-STATE CIRCUITS MAGAZINE

SU M M E R 2 0 2 0

IEEE Solid-States Circuits Magazine - Summer 2020

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Summer 2020