IEEE Circuits and Systems Magazine - Q3 2023 - 15

Table 3.
Performance comparison of various tiny models and inference frameworks on STM32H743, which runs at 480 MHz
with the resource constraint of 512 KB peak memory and 2 MB storage.
CMSIS-NN arXiv'18
[41]
(Continued)
X-Cube-AI [42]
Peak
Memory
Storage
usage
Peak
Memory
Storage
usage
Peak
Memory
Storage
usage
Peak
Memory
Storage
usage
OOM
OOM
516 ms
342 KB
TinyEngine
NeurIPS'20 [8]
463 ms
416 KB
TF-Lite Micro
MLSys'21 [47]
Dataset: ImageNet; Model: mcunet-in4; Input Resolution: 160; Width Multiplier: N/A; Top-1 Accuracy: 68.0%
Latency
OOM
OOM
OOM
54 ms
136 KB
1843 KB
35 ms
97 KB
1825 KB
23 ms
35 KB
OOM
Dataset: ImageNet; Model: proxyless-w0.3; Input Resolution: 64; Width Multiplier: 0.3; Top-1 Accuracy: 37.0%
Latency
512 ms
128 KB
1084 KB
380 ms
453 KB
865 KB
205 ms
221 KB
865 KB
29 ms
88 KB
777 KB
176 ms
259 KB
779 KB
23 ms
61 KB
1065 KB
Dataset: ImageNet; Model: proxyless-w0.3; Input Resolution: 176; Width Multiplier: 0.3; Top-1 Accuracy: 56.2%
Latency
3801 ms
453 KB
1084 KB
43 ms
173 KB
1065 KB
Dataset: ImageNet; Model: mbv2-w0.3; Input Resolution: 64; Width Multiplier: 0.3; Top-1 Accuracy: 34.1%
Latency
467 ms
173 KB
959 KB
768 KB
690 KB
940 KB
1All the inference frameworks used in this measurement are the latest versions as of December 19, 2022. 2The measurement of X-Cube-AI (v7.3.0) is with
the default compilation setting, i.e., balanced optimization. 3OOM denotes out of memory. 4All the models are available on https://github.com/mit-han-lab/
mcunet.
low memory consumption and high speed. MCUNetV2
identified that the imbalanced memory distribution is
the primary memory bottleneck in most convolutional
neural network designs, where the memory usage of the
first few blocks is an order of magnitude greater than
the rest of the network [9]. As a result, this study proposed
receptive field redistribution to shift the receptive
field and FLOPs to a later stage, reducing the halo's
computation overhead. To minimize the difficulty of
manually redistributing the receptive field, this study
also automated the neural architecture search process
to simultaneously optimize the neural architecture and
inference scheduling. UDC explored a broader design
search space to generate compressible neural networks
with high accuracy for neural processing units (NPUs),
which can address the memory problem by exploiting
model compression with a broader range of weight
quantization and sparsity [51].
THIRD QUARTER 2023
2) System Solutions
In recent years, popular training frameworks such as
PyTorch [89], TensorFlow [90], MXNet [91], and JAX [92]
have contributed to the success of deep learning. However,
these frameworks typically rely on a host language
(e.g., Python) and various runtime systems, which adds
significant overhead and makes them incompatible with
tiny edge devices. Emerging frameworks such as TVM
[93], TF-Lite [94], MNN [95], NCNN [96], TensorRT [97],
and OpenVino [98] offer lightweight runtime systems
for edge devices such as mobile phones, but they are
not yet small enough for MCUs. These frameworks cannot
accommodate IoT devices and MCUs with limited
memory.
CMSIS-NN implements optimized kernels to increase
inference speed, minimize memory footprint, and enhance
the energy efficiency of deep learning models
on ARM Cortex-M processors [41]. X-Cube-AI, designed
IEEE CIRCUITS AND SYSTEMS MAGAZINE
15
https://www.github.com/mit-han-lab/
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
