IEEE Circuits and Systems Magazine - Q3 2023 - 13

making without a cloud connection, achieving highaccuracy
and low-latency results for autonomous
driving scenarios [29], [30], [31].
■ Anomaly Detection: TinyML can equip robots and
sensors with the capability to perform anomaly
detection to reduce human efforts [32], [33], [34].
■ Ecology and Agriculture: TinyML can also help
with ecological, agricultural, environmental, and
phenomics applications so as to conserve endangered
species or forecast weather activities [35],
[36], [37], [38], [39], [40].
Overall, the potential applications of TinyML are diverse
and numerous, and will expand as the field continues
to advance.
II. Recent Progress in TinyML
A. Recent Progress on TinyML Inference
TinyML and deep learning on MCUs have seen rapid
growth in industry and academia in recent years. The
primary challenge of deploying deep learning models
on MCUs for inference is the limited memory and computation
available on these devices. For example, a popular
ARM Cortex-M7 MCU, the STM32F746, has only 320
KB of SRAM and 1 MB of flash memory. In deep learning
scenarios, SRAM limits the size of activations (read and
write) while flash memory limits the size of the model
(read-only). In addition, the STM32F746 has a processor
with a clock speed of 216 MHz, which is 10-20 times lower
than laptops. To enable deep learning inference on
MCUs, researchers have proposed various designs and
solutions to address these issues. Table 2 summarizes
the recent related studies on TinyML targeting MCUs, including
both algorithm solutions and system solutions.
In Table 3, we measured three different metrics (i.e.,
latency, peak memory, and flash usage) of four representative
related studies (i.e., CMSIS-NN [41], X-Cube-AI
[42], TinyEngine [8], and TF-Lite Micro [47]) on an identical
MCU (STM32H743) and identical datasets (VWW
and Imagenet), in order to provide a more accurate and
transparent comparison.
1) Algorithm Solutions
The importance of neural network's efficiency to the
overall performance of a deep learning system cannot
be overstated. Compressing off-the-shelf networks
by removing redundancy and reducing complexity
through pruning [57], [58], [59], [60], [61], [62] and quantization
[63], [64], [65], [66], [67], [68], [69], [70] are two
popular methods to improve network efficiency. Tensor
decomposition [71], [72], [73] is also an efficient
compression technique. In order to enhance network
THIRD QUARTER 2023
efficiency, knowledge distillation is also a method to
transfer information learned from one teacher model
to another student model [74], [75], [76], [77], [78], [79],
[80], [81]. Another method is to directly design tiny and
efficient network structures [4], [5], [6], [7]. Recently,
neural architecture search (NAS) has dominated the design
of efficient networks [82], [83], [84], [85], [86], [87].
To make deep learning feasible on MCUs, researchers
have proposed various algorithm solutions. Rusci et
al. [45] proposed a rule-based quantization strategy that
minimizes the bit precision of activations and weights
in order to reduce memory usage. Depending on the
memory constraints of various devices, this method
can quantize activations and weights with 8, 4, or 2
bits of mixed precision. On the other hand, although
neural architecture search (NAS) has been successful
in finding efficient network architectures, its effectiveness
is highly dependent on the quality of the search
space [88]. For MCUs with limited memory, standard
model designs and appropriate search spaces are especially
lacking. To address this, TinyNAS, proposed as
part of MCUNet, employs a two-step NAS strategy that
optimizes the search space according to the available
resources [8]. TinyNAS then specializes network architectures
within the optimized search space, allowing it
to automatically deal with a variety of constraints (e.g.,
device, latency, energy, memory) at low search costs.
MicroNets observed that the inference latency of networks
in the NAS search space for MCUs varies linearly
with the number of FLOPs in the model [48]. As a result,
it proposed differentiated NAS, which treats the
FLOPs as a proxy for latency in order to achieve both
Figure 2. We can't directly scale mobile ML or cloud ML models
for TinyML. MobilenetV2 [4] with a width of 1.4 was used
for the experiments. The batch size was set to 1 for inference
and 8 for training. While MobilenetV2 reduces the number of
parameters by 4.2× compared to ResNet, the peak memory
usage increases by 2.3× for inference and only improves by
1.1× for training. Additionally, the total required training memory
is 6.9× larger than the memory needed for inference.
These results demonstrate the significant memory bottleneck
for TinyML, and the bottleneck is the activation memory, not
the number of parameters.
IEEE CIRCUITS AND SYSTEMS MAGAZINE
13
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
