IEEE Circuits and Systems Magazine - Q3 2023 - 50

Therefore, these domain-specific accelerators also need
to be continuously upgraded, e.g., from the 28 nm TPUv1
in 2016 [13] to the 7 nm TPUv4 in 2021. Similar challenges
in high cost and limited lifespan still remain.
We identify a growing trend to emphasize on hardware
reuse and leverage advanced packaging to enhance
the capability of hardware systems. Using this
approach, a chip is designed to be a modular building
block, called chiplet; and a system is constructed by
reusing chiplets. To meet the requirements of different
DNN models and use cases, systems can be constructed
using the suitable number and types of chiplets. In other
words, this approach takes advantage of chiplet reuse to
construct systems in package (SiP). The premise of this
approach is that designing, fabricating and assembling
packages require lower cost and effort than designing
and fabricating large monolithic chips.
For the SiP approach to succeed, we identify three basic
requirements: 1) availability of reusable chiplets that are
equipped with high-bandwidth and efficient I/O interfaces;
2) accessible advanced packaging and assembly process;
and 3) methodology to map workloads to chiplet-based
systems. Among the three requirements, a high-bandwidth
and efficient I/O interface is necessary to ensure
that the chiplets that constitute an SiP can be seamlessly
integrated to match the performance of a monolithic chip;
an accessible advanced packaging and assembly process
ensure that high-density integration and high-bandwidth
routing are feasible to construct an SiP at a reasonable
cost; and a mapping methodology is needed to divide the
workload and assign them appropriately to the chiplets to
achieve high utilization and efficiency.
In the following, we use two recent designs as examples
to outline the primary ways in constructing SiP for
DNN compute acceleration. We categorize them into two
classes, homogeneous integration and heterogeneous
integration. In homogeneous integration, same chiplets
are tiled to scale up the system to support models of
larger size. In heterogeneous integration, different types
of chiplets are put together to extend the functionality
to cover new types of workloads.
A. Homogeneous Integration
The best example of homogeneous integration is Nvidia's
DNN multi-chip package (MCP) shown in Fig. 22,
where up to 36 DNN chiplets can be integrated in one
MCP to scale up the system as needed [52]. The DNN
chiplet measures 6 mm2 in a TSMC 16 nm technology.
It integrates tiles of SIMD-based PEs to provide up to
1,024 MACs/cycle (INT8) or 4 TOPS (INT8) [52].
Nvidia's DNN MCP is built on a 12-layer organic substrate.
Organic substrate is generally of lower cost than
substrates used for advanced packaging such as silicon
interposers, but the routing density is generally lower
too. Nvidia's DNN MCP adopts a serial link approach
to achieve a high inter-chiplet bandwidth using fewer
wires at very high speed, suitable for organic substrate.
In particular, the Nvidia design used a 200 mV lowswing,
short-reach serial link called ground-referenced
signaling (GRS) to achieve up to 25 Gbps/lane at
0.82-1.75 pJ/b for a short reach of 3-7 mm [53]. A chiplet
is equipped with four transmit lanes and four receive
lanes for up to 100 Gbps of input and 100 Gbps of output
bandwidth [52].
The compute and I/O specifications
Figure 22. Nvidia DNN MCP approach. Figure reused from [52] ©2020 IEEE.
50
IEEE CIRCUITS AND SYSTEMS MAGAZINE
above shed light on key design considerations
for a chiplet-based DNN accelerator:
1) the compute capacity of
the DNN chiplet (4 TOPS in INT8) significantly
exceeds the I/O bandwidth
(100 Gbps, transmit or receive) and
2) the compute energy efficiency of
the DNN chiplet (0.11 pJ/OP in INT8)
is substantially lower than the I/O energy
efficiency (0.82 pJ/b). The DNN
chiplet must reuse the input data (input
activations and weights) and reduce
the output data (output activations)
to minimize the I/O usage, or I/O can
easily overtake compute to become
the performance and energy bottleneck,
rendering the chiplet approach
impractical.
The contrast between compute
and I/O also has an implication on the
THIRD QUARTER 2023
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
