IEEE Circuits and Systems Magazine - Q3 2023 - 19

weights. It then performs an evolution search to find the
best model within the search space that meets the onboard
resource constraints while achieving the highest
accuracy. For each sampled network, it uses TinyEngine
to optimize the memory scheduling to measure the optimal
memory usage. With such a kind of co-design, we can
efficiently fit the tiny memory budget.
B. TinyEngine: A Memory-Efficient Inference Library
Researchers used to assume that using different deep
learning frameworks (libraries) will only affect the inference
speed but not the accuracy. However, this is not the
case for TinyML: the efficiency of the inference library
matters a lot to both the latency and accuracy of the
searched model. Specifically, a good inference framework
will make full use of the limited resources in MCUs, avoiding
waste of memory, and allowing a larger search space
for architecture search.
With a larger degree of design
freedom, TinyNAS is
more likely to find a highaccuracy
model. Thus, TinyNAS
is co-designed with
a memory-efficient inference
library, TinyEngine.
1) Code Generation
Most existing inference libraries
(e.g., TF-Lite Micro,
CMSIS-NN) are interpreter-based.
Though it is easy
to support cross-platform
development, it requires
extra memory, the most
expensive
resource
in
MCU, to store the meta-information
(such as model
structure parameters). Instead, TinyEngine only focuses
on MCU devices and adopts code generator-based
compilation. This not only avoids the time for runtime
interpretation, but also frees up the memory usage to allow
design and inference of larger models. Compared to
CMSIS-NN, TinyEngine reduced memory usage by 2.1×
and improve inference efficiency by 22% via code generation,
as shown in Figures 7 and 8.
The binary size of TinyEngine is lightweight, making it
very memory-efficient for MCUs. The model directly compiled
by well-known programming languages for deep
learning (e.g., Python, Cython, etc.) cannot be run on
MCUs as the size of their dependencies and packages are
already larger than the Flash size of MCUs, let alone the
size of the compiled model. Besides, unlike interpreterbased
TF-Lite Micro, which prepares the code for every
operation (e.g., conv, softmax) to support cross-model
Figure 6. Once-for-all [129] trains one single super network that supports a wide range of
sub-networks through weight sharing, and specializes different sub-network architectures for
different MCU hardware.
Figure 7. TinyEngine achieves higher inference efficiency than existing inference frameworks while reducing memory usage.
TinyEngine is up to 22×, 2.3×, and 1.5× faster than TF-Lite Micro, CMSIS-NN, and X-Cube-AI, respectively (left). By reducing
the memory usage, TinyEngine can run various model designs with tiny memory, enlarging the design space for TinyNAS under
the limited memory of MCU (right).
THIRD QUARTER 2023
IEEE CIRCUITS AND SYSTEMS MAGAZINE
19

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023