IEEE Circuits and Systems Magazine - Q3 2023 - 10

Table 1.
Microcontrollers have three orders of magnitude less memory and storage compared to mobile phones, and 5-6
orders of magnitude less than cloud GPUs (left). The extremely limited memory makes deep learning deployment
difficult. The peak memory and storage usage of widely used deep learning models (right). ResNet-50 exceeds the
resource limit on microcontrollers by 100×, MobileNet-V2 exceeds by 20×. Even the int 8 quantized MobileNetV2
requires 5.3× larger memory and can't fit a microcontroller.
ImageNet accuracy, MobileNetV2-1.4 has 4.2× smaller
model size compared to ResNet-50, but its peak memory
even larger by 2.3×. Using MobileNet designs does not
adequately address the SRAM limit, instead, it actually
makes the situation even worse compared to ResNet.
Therefore, we need to rethink the model design principles
for TinyML.
2) Directly Adapting Models for Inference
Does Not Work for Tiny Training
Training poses an even greater challenge in terms of resource
constraints, as intermediate activations must be
stored in order to compute backward gradients. When
moving from inference to training with full backpropagation,
the required memory increases by a factor of
6.9. As shown in Figure 2 the training memory requirements
of MobileNets are not much better than ResNets
(improved by only 10%). Tiny IoT devices such as microcontrollers
typically have a limited SRAM size, such
as 256 KB, which is barely enough for the inference of
deep learning models, let alone training. Previous work
in the cloud and mobile AI has focused on reducing
FLOPs [4], [5], [7] or only optimizing inference memory
[8], [9]. However, even using memory-efficient inference
models such as MCUNet [8] to bridge the three orders
of magnitude gap, training is still too expensive for tiny
platforms. If we follow conventional full model update
schemes, the model must be scaled down significantly
to fit within the tight memory constraints, resulting
in low accuracy. This highlights the need to redesign
backpropagation schemes and investigate new learning
algorithms to reduce the main activation memory
bottleneck and enable fast and accurate training on tiny
devices. In Section IV, we will discuss this issue in detail
and introduce the concept of sparse layer and sparse
tensor updates.
3) Co-Design is Necessary for TinyML
Co-design is necessary for TinyML because it allows us
to fully customize the solutions that are optimized for
10
IEEE CIRCUITS AND SYSTEMS MAGAZINE
the unique constraints of tiny devices. Previous neural
architectures like MobileNets [4], [5], and ResNets [10]
are designed for mobile/cloud scenarios but not wellsuited
for tiny hardware. Therefore, we need to design
neural architectures that are suitable for TinyML applications.
On the other hand, existing deep training frameworks
are optimized for cloud servers and lack support
for memory-efficient forward and backward, thus cannot
fit into tiny devices. The huge gap (> 1000 ×) between
the resources of tiny IoT devices and the requirements
of current frameworks prohibits the usage. To address
these challenges, it is necessary to develop algorithms,
systems, and training techniques that are specifically
tailored to the settings of these tiny platforms.
B. Applications of TinyML
By democratizing costly deep learning models to IoT
devices, TinyML has many practical applications. Some
example applications include:
■ Personalized Healthcare: TinyML can allow wearable
devices, such as smartwatches, to continuously
track the activities and oxygen saturation status
of the user in order to provide health suggestions
[11], [12], [13], [14]. Body pose estimation is also a
crucial application for elderly healthcare [15].
■ Wearable Applications: TinyML can assist people
with wearable or IoT devices for speech applications,
e.g., keyword spotting, automatic speech
recognition, and speaker verification [16], [17], [18].
■ Smart Home: TinyML can enable object detection,
image recognition, and face detection on IoT devices
to build smart environments, such as smart
homes and hospitals [19], [20], [21], [22], [23].
Interface: TinyML can enable
■ Human-Machine
human-machine interface applications, like hand
gesture recognition [24], [25], [26], [27]. TinyML
is also capable of predicting and recognizing sign
languages [28].
■ Smart Vehicle and Transportation: TinyML can perform
object detection, lane detection, and decision
THIRD QUARTER 2023

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023