IEEE Solid-States Circuits Magazine - Summer 2020 - 32

instruction bookkeeping overhead
by using large aggregate instructions
(e.g., single-instruction, multiple-data
(SIMD), vector instructions, single-instruction, multiple-threads (SIMT), or
tensor instructions), where a single instruction can be used to initiate multiple operations.
The number of PEs and the peak
throughput of a single PE indicate
on ly t he t he or et ic a l m a x imu m
throughput (i.e., peak performance)
when all PEs are performing computation (100% utilization). In reality,
the achievable throughput depends
on the actual utilization of those PEs,
which is affected by several factors
as follows:
utilization of PEs
= number of active PEs
number of PEs
# utilization of active PEs. (3)

Performance (Operations/s)

The first term reflects the ability to distribute the workload to PEs,
while the second term reflects how
efficiently those active PEs are processing the workload. The number of
active PEs is the number of PEs that
receive work (the ratio of active PEs
to the total number of PEs can be referred to as the active PE percentage);
therefore, it is desirable to distribute
the workload to as many PEs as possi-

ble. The ability to distribute the workload is determined by the flexibility
of the architecture, for instance, the
on-chip network, to support the different layer shapes in a DNN model as
explored in [9] and [10].
Within the constraints of the onchip network, the number of active
PEs is also determined by the specific
allocation of work to PEs by the mapping process. The mapping process
involves the placement and scheduling in space and time of every MAC
operation (including the delivery of
the appropriate operands) onto the
PEs. The mapper can be thought of
as a compiler for the DNN processor
[11]. The mapping process, on a layerby-layer basis, is explored in detail
in [12]-[14]. Additional challenges regarding the flexibility of mapping are
discussed in the "Energy Efficiency
and Power Consumption" section.
The utilization of active PEs is
largely dictated by the timely delivery of work to the PEs such that the
active PEs do not become idle while
waiting for the data to arrive. This
can be affected by the bandwidth
(BW) and latency of the (on-chip and
off-chip) memory and network. The
BW requirements can be affected by
the amount of data reuse available
in the DNN model and the amount
of data reuse that can be exploited

Slope = BW
Inflection Point
Peak
Performance

BW
Limited
Optimal
Operational
Intensity

Computation
Limited
Operational Intensity
(Operations/Byte)

FIGURE 5: The roofline model. The peak operations per second is indicated by the bold line;
when the operation intensity (dictated by the amount of compute per byte of data) is low,
the operations per second is limited by the data delivery. The design goal is to operate as
closely as possible to the peak operations per second for the operation intensity of a given
workload.

SU M M E R 2 0 2 0

IEEE SOLID-STATE CIRCUITS MAGAZINE

by the memory hierarchy and dataflow. The dataflow determines the
order of operations and where data
are stored and reused. The amount
of data reuse can also be increased
using a larger batch size, which is
one of the reasons that increasing
batch size can increase throughput.
The challenges of data delivery and
memory BW are discussed in [14] and
[15]. The utilization of active PEs can
also be affected by the imbalance
of work allocated across PEs, which
may occur when exploiting sparsity
(i.e., avoiding unnecessary work associated with multiplications by
zero); PEs with less work become idle
and, thus, have lower utilization.
There is also an interplay between
the number of PEs and the utilization
of PEs. For instance, one way to reduce
the likelihood that a PE needs to wait
for data is to store some data locally
near or within the PE. However, this
requires increasing the chip area
allocated to on-chip storage, which,
given a fixed chip area, would reduce
the number of PEs. Therefore, a key
design consideration is the area allocation between compute (which
increases the number of PEs) versus
on-chip storage (which increases the
utilization of PEs).
The impact of these factors can be
captured using Eyexam, a systematic
way of understanding the performance limits for DNN processors as
a function of specific characteristics
of the DNN model and processor design. Eyexam includes and extends
the well-known roofline model [16].
The roofline model, as illustrated in
Figure 5, relates average BW demand
and peak computational ability to
performance.
The goal of Eyexam is to provide a
fine-grained performance profile for
a DNN processor. It is a sequential
analysis process that involves seven
major steps, as shown in Figure 6. The
process starts with the assumption that
the architecture has infinite processing
parallelism, storage capacity, and data
BW. Therefore, it has infinite performance (as measured in MAC operations
per cycle).

IEEE Solid-States Circuits Magazine - Summer 2020

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Summer 2020