IEEE Solid-State Circuits Magazine - Fall 2017 - 64

Recent work has shown that the combination
of pruning, weight sharing, and Huffman
compression compresses state-of-the-art
networks by 50 times in memory size.
decompress the data and, at best,
remain idle during zero-valued operations. The efficient-inference engine
[35], however, demonstrates that it
is also possible and highly beneficial to operate directly on the compressed data by adapting the data
path and memory interface to the
compressed data format.
A network compression technique
that does enable straightforward
network execution in the complex
domain without any hardware adaptation uses singular value decomposition (SVD) [36]. By performing SVD on
a sparse weight matrix of a fully connected network layer, the matrix can
be decomposed into two matrices,
the rows and columns of which are
ordered by the function of the most
significant network parameters. By
simply removing the nonsignificant
sections of the matrix, one is left with
a strongly compressed representation of the original network layer. The
result can be executed on any regular neural network accelerator, as it
is identical to the execution of two
(much smaller) fully connected layers.
While this method is more straightforward from a hardware point of view, it

In this short tutorial, we have presented a selection of very promising
hardware and algorithmic techniques
from the rapidly expanding and
growing field of deep learning. Each
exploits and/or enhances the unique
features of deep networks to improve
the energy efficiency of their execution. Together, they have allowed the
achievement of tremendous energy
savings compared to traditional
CPU- and GPU-based compute platforms. As can be seen in Figure 12
[37], this recent wave of innovations
breaks the barrier for embedded
deep inference in mobile devices.
Implementations far surpassing the
efficiencies of 1-TOP/W have recently
been demonstrated, while computational throughput is boosted to several 100 GOP.
Still, challenges remain to effectively bring deep learning to IoT and
edge devices. First, few (if any) complete end-to-end solutions have been
demonstrated. Doing so involves

100 GOPs

Energy-Efficiency (TOPs/W)

Outlook

References

0.1

offers only limited compression capabilities, ranging typically up to only
five times compression [36].

10-f/s ResNet at 30 mW

1 TOPs/W

GPU

CPU
1

100

4b
4-b Sparse
8b
16 b
Minimum Energy
Peak Performance
2016 References

1,000

Throughput (GOPs)
FIGURE 12: An overview of the reported performance of the deep neural network processors
published at the International Solid-State Circuits Conference in 2016 and 2017. Performances
beyond 100 GOP and 1 TOP/W will be a game changer for deep inference in embedded devices.

integrating the deep-inference chips
in complete vision-processing pipelines mapping real-life applications.
This requires not only an efficient
execution of the inference kernel itself
but also efficient image slicing, data
transfer, and results interpretation.
A second interesting challenge
lies in the learning process. So far, most
chips focus on the inference part,
where pretrained models are efficiently
executed on-chip. In the future, however, the desire for more privacy and
user customization will stimulate chips
capable of executing the training phase
as well. This, however, comes with new
computational challenges and the
need for a careful algorithm-architecture cooptimization.
It is, thus, very clear that, more
than ever, the hardware and algorithmic layer must be optimized
jointly, grasping the various crosslayer opportunities of deep neural
networks. This is also apparent from
the interest of many traditionally
software-oriented companies (like
Google, Amazon, and Microsoft) in
the development of new proprietary
hardware for deep learning.
This field is so vibrant that every
single week new ideas pop up. Of
course, space does not allow us to
cover all of the exciting ideas going
around in the embedded deep learning space at the moment. Yet we hope
that we were able to spark readers'
interest and stimulate further exploration of this lively field.

FA L L 2 0 17

IEEE SOLID-STATE CIRCUITS MAGAZINE

[1] Y. LeCun, Y. Bengio, and G. Hinton, "Deep
learning," Nature, vol. 521, no. 7553, pp
436-444 2015.
[2] O. Russakovsky, J. Deng, H. Su, J. Krause,
S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, A. C. Berg, and
L. Fei-Fei, "ImageNet large scale visual
recognition challenge," Int. J.Computer
Vision, vol. 115, no. 3, pp. 211-252,
2015.
[3] A. Krizhevsky, I. Sutskever, and G. Hinton,
"ImageNet classification with deep convolutional neural networks," in Proc. Conf.
Neural Information Processing Systems,
2012, pp. 1097-1105.
[4] K. He, X. Zhang, S. Ren, and J. Sun, "Deep
residual learning for image recognition," arXiv Preprint, arXiv:1512.03385,
2015.
[5] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S.
Reed, D. Anguelov, and A. Rabinovich,
"Going deeper with convolutions," in Proc.

Table of Contents for the Digital Edition of IEEE Solid-State Circuits Magazine - Fall 2017