IEEE Solid-States Circuits Magazine - Summer 2020 - 30

present equations that describe
the relationship between the factors and the metrics
■■ describe how these metrics can be
incorporated into design considerations for both the DNN hardware and the DNN model
■■ specify what should be reported
for a given metric to enable proper
evaluation.
Finally, we highlight tools that can
be used to evaluate some of these
metrics early in the design process
(to enable rapid design exploration)
and provide a case study on how
one might bring all of these metrics
together for a holistic evaluation of
a given approach. First, however, we
discuss each of the metrics.

Accuracy

DRAM

Global Buffer

Accuracy indicates the quality of the
result for a given task. The fact that

DNNs can achieve state-of-the-art accuracy on a wide range of tasks is one
of the key reasons driving their popularity and wide use today. The units
used to measure accuracy depend on
the task. For instance, for image classification, accuracy is reported as the
percentage of correctly classified
images, while, for object detection,
accuracy is reported as the mean average precision, which is related to the
tradeoff between true positives, false
positives, and false negatives.
Factors that affect accuracy include
the difficulty of the task and data
set. (Ideally, robustness and fairness
should be considered in conjunction
with accuracy, as there is also an interplay between these factors; however,
these are areas of ongoing research
and beyond the scope of this article.)
For instance, classification on the ImageNet data set [2] is much more dif-

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

FIGURE 3: The typical hardware architecture of a DNN processor.

(a)

ficult than on the MNIST data set [3]
(Figure 4), and object detection is usually more difficult than classification.
As a result, a DNN model that performs
well on MNIST may not necessarily perform well on ImageNet. Achieving high
accuracy on difficult tasks or data sets
typically requires more complex DNN
models (e.g., a larger number of MAC
operations and more distinct weights,
increased diversity in layer shapes,
and so on), which can impact how efficiently the hardware can process the
DNN model.
Accuracy should, therefore, be
interpreted in the context of the difficulty of the task and data set. (As
an analogy, getting nine out of 10 answers correct on a high school exam
is different than nine out of 10 answers correct on a college-level exam.
One must look beyond the score and
consider the difficulty of the exam.)
Evaluating hardware using well-studied, widely used DNN models, tasks,
and data sets can allow one to better
interpret the significance of the accuracy metric.
Recently, motivated by the impact
of the SPEC benchmarks for general
purpose computing [4], several industry and academic organizations have
put together a broad suite of models,
called MLPerf, to serve as a common
set of well-studied DNN models to
evaluate the performance and enable
fair comparison of various software
frameworks, hardware architectures,
and cloud platforms for both training

(b)

FIGURE 4: The (a) MNIST data set (10 classes, 60,000 training, and 10,000 testing) [3] versus the (b) ImageNet data set (1,000 classes,
1.3 million training, and 100,000 testing) [2].

30	

SU M M E R 2 0 2 0	

IEEE SOLID-STATE CIRCUITS MAGAZINE
IEEE Solid-States Circuits Magazine - Summer 2020

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Summer 2020
