IEEE Solid-States Circuits Magazine - Fall 2022 - 26

channels in the beginning of the network
and smaller feature maps with
a larger number of channels in layers
toward the end. As a result, the deeper
layers can maximally exploit the wide
AiMC parallelism, while the early layers
often cannot fully utilize the AiMC
core and map more efficiently on the
more flexible digital accelerator. Also,
other layers, such as depthwise kernels
or fully connected layers, require
the flexibility of the digital core for efficient
execution.
Figure 6(c) shows a resulting
mapping of the ResNet layers of Figure
6(b) on the different ML processor
cores of Diana, together with the
data dependencies of the tiles across
the layers. As visualized, an important
added benefit of the multi-core
setup is that the different cores can
operate in parallel, feeding each other
data through a shared L1 buffer under
a depth-first processing scheme.
For Diana, such depth-first operation
brought 40% latency benefits and a
more than 7× lower on-chip memory
footprint over the layer-wise execution
scheme [Figure 6(d)]. One could
think of alternative heterogeneous
architectures with more cores and
which, e.g., combine sparse and dense
compute cores or depthwise- versus
pointwise-optimized cores.
Hence, similar to heterogeneous
multi-core CPUs, heterogeneous
multi-core ML accelerators can enable
drastic efficiency improvements, yet
they also come with several challenges.
To fully exploit the potential
benefits stemming from such heterogeneous
multi-core architectures,
it
is important to keep all cores usefully
busy across the wide workload variety.
As not all NNs make use of the
same layer types, not all specialized
cores will always be equally useful.
This can lead to core underutilization
and load unbalancing. When
silicon area is abundantly available,
this does not have to be a problem,
resulting in dark silicon. Yet in costconstrained
extreme edge platforms,
this is typically not the case.
Heterogeneity is of course already
strongly present at the system level
26
FALL 2022
in edge platforms, with GPUs, CPUs,
and neural processing units (NPUs)
combined in nearly any embedded
system. It is still to be seen to what extent
the heterogeneity will penetrate
into the NPU itself. At the moment,
it is not clear yet for which networks
and layer types heterogeneous cores
are actually beneficial and what the
optimum core combination would
look like. Emerging ML models, such
as transformers, logic reasoning,
symbolic artificial intelligence (AI), or
probabilistic graphical models, might
even lead to completely new processor
classes to be integrated together
with the CPUs, GPUs, and NPUs. One
example of this is our recent dataflow
graph (DAG) processing unit [27],
optimized for evaluating irregular
sparse directed acyclic DAGs.
So, the research debate is open.
Without any doubt, the future of ML
processors will be multi-core, either
integrated on a single chip or integrated
on interposers using chiplets. The
hardware-scheduler design space
is very large, but the potential gains of
all scheduling opportunities among a
set of homogeneous or heterogeneous
cores are very large, too. So, a wide
research field is still open on optimal
core architectures, multi-core specialized
schedulers, and heterogeneous
ML compilers. There is a strong need
for hardware-aware design space exploration
frameworks, such as our
ZigZag tool and its multi-core/layerfusion
enhanced successor Stream.
The future will be filled with many
more architecture and scheduling research
challenges ahead. We look forward
to these exciting times for ML
hardware research. The future is bright.
Acknowledgment
This work was supported in part by
KU Leuven, the Flemish Government
(AI Research Program, FlandersAI),
and the Fund for Scientific Research
Flanders (FWO-Vlaanderen) under
Grant G006718N.
References
[1] A. Agrawal et al., " A 7nm 4-core AI chip
with 25.6TFLOPS hybrid FP8 training,
102.4TOPS INT4 inference and workloadIEEE
SOLID-STATE CIRCUITS MAGAZINE
aware throttling, " in Proc. 2021 IEEE Int.
Solid-State Circuits Conf. (ISSCC), vol. 64,
pp. 144-146, doi: 10.1109/ISSCC42613.
2021.9365791.
[2] G. W. Burr, S. Lim, B. Murmann, R. Venkatesan,
and M. Verhelst, " Fair and comprehensive
benchmarking of machine learning
processing chips, " IEEE Des. Test., vol. 39,
no. 3, pp. 18-27, Jun. 2022, doi: 10.1109/
MDAT.2021.3063366.
[3] V. Camus, L. Mei, C. Enz, and M. Verhelst,
" Review and benchmarking of precisionscalable
multiply-accumulate unit architectures
for embedded neural-network
processing, " IEEE J. Emerg. Sel. Topics Circuits
Syst., vol. 9, no. 4, pp. 697-711, Dec.
2019, doi: 10.1109/JETCAS.2019.2950386.
[4] Y.-H. Chen, T. Krishna, J. Emer, and V.
Sze, " Eyeriss: An energy-efficient reconfigurable
accelerator for deep convolutional
neural networks, "
[5] Y. Chen, T. Chen, Z. Xu, N. Sun, and O.
Temam, " DianNao family: Energy-efficient
hardware accelerators for machine learning, "
Commun. ACM, vol. 59, no. 11, pp.
105-112, Oct. 2016, doi: 10.1145/2996864.
" A heterogeneous in[6]
A. Garofalo et al.,
memory computing cluster for flexible endto-end
inference of real-world deep neural
networks, " IEEE J. Emerg. Sel. Topics Circuits
Syst., vol. 12, no. 2, pp. 422-435, Jun.
2022, doi: 10.1109/JETCAS.2022.3170152.
[7] M. Giordano et al., " CHIMERA: A 0.92
TOPS, 2.2 TOPS/W edge AI accelerator
with 2 MByte on-chip foundry resistive
RAM for efficient training and inference, "
in Proc. 2021 Symp. VLSI Circuits,
pp. 1-2, doi: 10.23919/VLSICircuits52068.
2021.9492347.
[8] K. Goetschalckx and M. Verhelst, " Breaking
high-resolution CNN bandwidth barriers
with enhanced depth-first execution, " IEEE
J. Emerg. Sel. Topics Circuits Syst., vol. 9,
no. 2, pp. 323-331, Jun. 2019, doi: 10.1109/
JETCAS.2019.2905361.
[9] K. Goetschalckx, and M. Verhelst, " DepFiN:
A 12nm, 3.8TOPs depth-first CNN processor
for high res. image processing, " in Proc. 2021
Symp. VLSI Circuits, pp. 1-2, doi: 10.23919/
VLSICircuits52068.2021.9492338.
[10] K. Hegde, P.-A. Tsai, S. Huang, V. Chandra,
A. Parashar, and C. W. Fletcher, " Mind
mappings: Enabling efficient algorithmaccelerator
mapping space search, " in
Proc. 26th ACM Int. Conf. Archit. Support
Program. Lang. Oper. Syst., ser. ASPLOS.
New York, NY, USA: Association for
Computing Machinery, 2021, pp. 943-958,
doi: 10.1145/3445814.3446762.
[11] P. Houshmand et al., " Opportunities
and limitations of emerging Analog inmemory
compute DNN architectures, "
in Proc. 2020 IEEE Int. Electron Devices
Meeting (IEDM), pp. 29-31, doi: 10.1109/
IEDM13553.2020.9372006.
[12] E. M. Ibrahim, L. Mei, and M. Verhelst, " Taxonomy
and benchmarking of precisionscalable
MAC arrays under enhanced DNN
dataflow representation, " IEEE Trans. Circuits
Syst. I, Reg. Papers, vol. 69, no. 5, pp.
2013-2024, May 2022, doi: 10.1109/TCSI.
2022.3141519.
[13] V. Jain, S. Giraldo, J. De Roose, B. Boons,
L. Mei, and M. Verhelst, " TinyVers: A 0.8-17
TOPS/W, 1.7 μW-20 mW, tiny versatile system-on-chip
with state-retentive eMRAM
for machine learning inference at the extreme
edge, " in Proc. 2022 IEEE Symp. VLSI
Technol. Circuits, pp. 20-21, doi: 10.1109/
VLSITechnologyandCir46769.2022.9830409.
in Proc. 2016
IEEE Int. Solid-State Circuits Conf. (ISSCC),
pp. 262-263, doi: 10.1109/ISSCC. 2016.
7418007.
IEEE Solid-States Circuits Magazine - Fall 2022

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2022
