IEEE Circuits and Systems Magazine - Q3 2023 - 31

compared to the full update thanks to the graph pruning
mechanism, while achieving the same or higher transfer
learning accuracy (compare the data points connected
by arrows in Figure 18). The memory is further reduced
with operator reordering, leading to 20-21× total memory
saving. With both techniques, the training of all three
models fits 256 KB SRAM.
The training latency per image on the STM32F746 MCU
is measured in Figure 21(c). By graph optimization and exploiting
multiple compiler optimization approaches (such
as loop unrolling and tiling), our sparse update + TTE kernels
can significantly enhance the training speed by 23−25×
compared to the full update + TF-Lite Micro kernels, leading
to energy saving and making training practical.
V. Conclusion and Outlook
In conclusion, TinyML is a rapidly evolving field that enables
deep learning on resource-constrained devices.
It fosters a wide range of customized and private AI applications
on edge devices, which can process the data
collected from the sensors right at the source. We point
out several unique challenges of TinyML. First, we need
to redesign the model design space since deep models
designed for mobile and other platforms do not work well
for TinyML. Second, we need to redesign backpropagation
schemes and investigate new learning algorithms
since directly adapting models for inference does not
work for tiny training. Third, co-design is necessary for
TinyML. We summarize the related works aiming to overcome
the challenges from the algorithm and the system
perspectives. Furthermore, we introduce the TinyML
techniques that not only enable practical AI applications
on a wide range of IoT platforms for inference, but also allow
AI to be continuously trained over time, adapting to a
world that is changing fast. Looking to the future, TinyML
will continue to be an active and rapidly growing area,
which requires continued efforts to improve the performance
and energy efficiency. We discuss several possible
directions for the future development of TinyML.
A. More Applications and Modalities
This review mainly focuses on convolutional neural networks
(CNNs) as computer vision is widely adapted to tiny
devices. However, TinyML has a broad range of applications
beyond computer vision, including but not limited to audio
processing, language processing, anomaly detection, etc.,
with sensor inputs from temperature/humidity sensors,
accelerometers, current/voltage sensors, among others.
TinyML enables local devices to process multiple-sensor
inputs to handle multi-task workloads, opening up future
avenues for numerous potential applications. We will leave
further exploration of these possibilities for future work.
THIRD QUARTER 2023
B. Self-Supervised Learning
Obtaining accurately labeled data for on-device learning
on the edge can be challenging. In some cases, like
keyboard typing, we can use the next input word as the
prediction target for the model. However, this is not
always practical for most applications, such as domain
adaptation for vision tasks (e.g., segmentation, detection),
obtaining supervision can be expensive and difficult.
One potential solution is to design self-supervised
learning tasks for on-device training, as has been proposed
in recent research [151].
C. Relationship Between TinyML and LargeML
TinyML and LargeML both aim to develop efficient models
under resource constraints such as memory, computation,
engineering effort, and data. While TinyML is
primarily focusing on making models run efficiently on
small devices, many of its techniques can also be applied
in cloud environments for large-scale machine learning
scenarios. For example, quantization techniques have
been effective in both TinyML [8], [9] and LargeML settings
[152], [153], and the concept of sparse learning has
been used in both scenarios to run models efficiently
with limited resources [56], [154]. These efficient techniques
are generally applicable and should not be limited
to TinyML settings.
The concept of TinyML is constantly evolving and
expanding. When ResNet-50 [10] was first introduced
in 2016, it was considered as a large model with 25 M
parameters and 4G MACs. However, six years later, with
the rapid advances in hardware, it can now achieve submillisecond
inference on a smartphone DSP (Qualcomm
Snapdragon 8Gen1). As hardware continues to improve,
what was once considered a " large " model may be considered
" tiny " in the future. The scope of TinyML should
evolve and adapt over time.
VI. Acknowledgment
We thank MIT AI Hardware Program, National Science
Foundation, NVIDIA Academic Partnership Award, MITIBM
Watson AI Lab, Amazon and MIT Science Hub, Qualcomm
Innovation Fellowship, and Microsoft Turing Academic
Program for supporting this research.
Ji Lin is currently pursuing the Ph.D.
degree with MIT EECS, Cambridge,
MA, USA. Previously, he graduated
from the Department of Electronic Engineering,
Tsinghua University, Beijing,
China. His research interests lie in efficient
and hardware-friendly machine learning, model
compression, and acceleration.
IEEE CIRCUITS AND SYSTEMS MAGAZINE
31
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
