IEEE Circuits and Systems Magazine - Q3 2023 - 9

M
I. Overview of Tiny Machine Learning
achine learning (ML) has made significant
impacts on various fields, including vision,
language, and audio. However, state-of-theart
models often come at the cost of high computation
and memory, making them expensive to deploy. To address
this, researchers have been working on efficient
algorithms, systems, and hardware to reduce the cost
of machine learning models in various deployment scenarios.
There are two main subdomains of efficient ML:
EdgeML and CloudML (Figure 1). While CloudML focuses
on improving latency and throughput on cloud servers,
EdgeML focuses on improving energy efficiency, latency,
and privacy on edge devices. These two domains
also intersect in areas such as hybrid inference [1],
[2], over-the-air (OTA) updates, and federated learning
between the edge and cloud [3]. In recent years, there
has been significant progress in extending the scope of
EdgeML to ultra-low-power devices such as IoT devices
and microcontrollers, known as TinyML.
TinyML has several key advantages. It enables machine
learning using only a few hundred kilobytes of
memory which greatly reduces the cost. With billions of
IoT devices producing more and more data in our daily
lives, there is a growing need for low-power, always-on,
on-device AI. By performing on-device inference near
the sensor, TinyML enables better responsiveness and
privacy while reducing the energy cost associated with
wireless communication. On-device processing of data
can be beneficial for applications where real-time decision-making
is crucial, such as autonomous vehicles.
In addition to inference, we push the frontier of TinyML
to enable on-device training on IoT devices. Itrevolutionizes
EdgeAI through continuous and lifelong
learning. Edge device can finetune the model on itself
rather than transmitting data to cloud servers, which
protects privacy. On-device learning has numerous benefits
and a variety of applications. For example, home
cameras can continuously recognize new faces, and
email clients can gradually improve their prediction by
updating customized language models. It also enables
IoT applications that do not have a physical connection
to the internet to adapt to the environment, such as precision
agriculture and ocean sensing.
In this review, we will first discuss the definition and
challenges of TinyML, analyzing why we can't directly
scale mobile ML or cloud ML models for tinyML. Then
we delve into the importance of system-algorithm codesign
in TinyML. We will then survey recent literature
and the progress of the field, presenting a holistic survey
and comparison in Tables 2 and 3. Next, we will introduce
our TinyML project, MCUNet, which combines
efficient system and algorithm design to enable TinyML
for both inference to training. Finally, we will discuss
several emerging topics for future research directions
in the field.
A. Challenges of TinyML
The success of deep learning models often comes at the
cost of high computation, which is not feasible for use
in TinyML applications due to the strict resource constraints
of devices such as microcontrollers. Deploying
and training AI models on MCU is extremely hard: No
DRAM, no operating systems (OS), and strict memory
constraints (SRAM is smaller than 256 kB, and FLASH
is read-only). The available resources on these devices
are orders of magnitude smaller than those available on
mobile platforms (see Table 1). Previous work in the field
has either (I) focused on reducing model parameters
without addressing the real bottleneck of activations or
(II) only optimized operator kernels without considering
improving the network architecture design. Neither of
which considers the problem from a co-design perspective,
and this has led to less optimal solutions for TinyML
applications. We observe several unique challenges of
TinyML and postulate how they might be overcome:
Figure 1. Efficiency is critical
CloudML
TinyML.
targets
for CloudML, EdgeML, and
high-throughput
accelerators
like GPUs, while EdgeML focuses on portable devices like
mobile phones. TinyML further pushes the efficiency boundary,
enabling powerful ML models to run on ultra-low-power
devices such as microcontrollers.
1) Models Designed for Mobile Platforms
Does Not Fit TinyML
There has been a lot of effort optimizing deep learning
models for mobile platforms like MobileNets [4], [5]
and ShuffleNet [6]. However, since mobile devices have
sufficient memory resources (Table 2), the model designs
focus on parameters/FLOPs/latency reduction but
not peak memory usage. As shown in Figure 2 left and
middle, comparing two models with the same level of
The authors are with the Massachusetts Institute of Technology, Cambridge, MA 02139 USA (e-mail: songhan@mit.edu).
THIRD QUARTER 2023
IEEE CIRCUITS AND SYSTEMS MAGAZINE
9

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023