IEEE Solid-States Circuits Magazine - Fall 2022 - 23

execution of multiple layers in a
network-or even multiple networks
across concurrent tasks-as
an additional spatial unrolling parameter.
Examples are layer 1 and
layer 2 in in Figure 4 (bottom left).
Each layer of each network can now
individually be scheduled on a (set
of) core(s). These increased degrees
of freedom for the scheduler improve
not only spatial utilization
but also temporal utilization as
temporal stalls due to data transfers
from/to memory can be hidden
underneath other computes.
To further alleviate the memory
needs and I/O bandwidth requirements
of such widely parallel execution,
the multi-core processing can be
combined with the aforementioned
depth-first (layer-fused) scheduling,
as shown in Figure 4 (bottom right).
This fine-grain scheduling allows the
cores to start the execution of layer N
+ 1 of a network before the execution
for layer N has fully finished, resulting
in lower latency and lower memory
footprints. In the example of Figure 4
(bottom right), layer 1 and layer 2 can
start immediately on core 2 and core 3
after the first tile's results of layer 0
are generated by core 0 and core 1.
It is clear that the migration to
multi-core architectures drastically
enlarges the scheduling space, now
enabling one to achieve increased
throughput without sacrificing
MAC utilization. This new processing
paradigm, however, also drastically
increases the hardware design
space. A wide variety of multi-core
architectures have been proposed
[1], [6], [7], [16], [19], [28], [34], [36].
They can first of all be categorized
in terms of the granularity at which
this additional degree of parallelization
is added (Figure 5). A first set of
parallel processors combines multiple
parallel SIMD compute arrays,
each equipped only with small local
memories, while still sharing all remaining
memories and control (e.g.,
within one neural processing unit
engine of [14]). Such a scheme allows
one to more flexibly map a single NN
layer with reconfigurable spatial unrollings
across the subcores but lacks
the flexibility to map multiple layers
in parallel.
Alternative architectures effectively
combine individual cores, each
with their own memory hierarchy and
control [6], [16], [19], [34], [36]. These
cores operate autonomously, each
running their own workload, while
the shared data can be communicated
Layer-By-Layer Scheduling
Depth-First Scheduling
Hardware Mapping:
(a) Single-Core NPU
(b) Multi-Core NPU
Layer 0
Layer 1Layer 2
Layer 3
Layer 4
Underutilized
Peak Performance
Core0
Core0
L0
L1 L2
(a)
Increase Computation Parallelism
Most Memory and Latency Savings
Core0 Core1
Core2 Core3
(b)
FIGURE 4: The impact of migrating from layer-by-layer scheduling versus depth-first scheduling (horizontal), respectively from single-core to
multi-core operations (vertical). (a) Single-core NPU. (b) Multi-core NPU.
IEEE SOLID-STATE CIRCUITS MAGAZINE
FALL 2022
23
Core0
Core1
Core2
Core3
L0
L1
L4
L2 L3
Core0
Core1
Core2
Core3
L3
Time
L4
Core0
Layer Tiling
One
Fused
Stack
Reduce Memory Demands
Small Tiles Worsen Utilization
IEEE Solid-States Circuits Magazine - Fall 2022

Table of Contents for the Digital Edition of IEEE Solid-States Circuits Magazine - Fall 2022
