IEEE Circuits and Systems Magazine - Q3 2023 - 20

Figure 8. TinyEngine outperforms existing libraries by eliminating runtime overheads, specializing each optimization technique,
and adopting in-place depth-wise convolution. This effectively enlarges design space for TinyNAS under a given latency/
memory constraint.
3) Patched-Based Inference
TinyNAS and TinyEngine have significantly reduced
the peak memory at the same level of accuracy. But
we still notice a very imbalanced peak memory usage
per block.
Imbalanced Memory Distribution. As an example,
Figure 9. Binary size comparison between different frameworks
(TF-Lite Micro[47], CMSIS-NN[41], and TinyEngine[8])
and models (SmallCifar[8] and MbV2[4]) during deployment.
We scale the width multiplier and input resolution of MbV2 to
0.35 and 64 so that most libraries can fit the neural network.
inference even if they are not used, which has high redundancy.
TinyEngine only compiles the operations that
are used by a given model into the binary. That is, the
reduction of binary size of the model compiled by TinyEngine
comes from not only the benefit of compilation over
interpretation but also the model-specific optimization/
specialization. As shown in Figure 9, such model-adaptive
compilation reduces code size by up to 4.5× and 5.0× compared
to TF-Lite Micro and CMSIS-NN, respectively.
2) In-Place Depth-Wise Convolution
TinyEngine supports in-place depth-wise convolution to
further reduce peak memory. Different from standard
convolutions, depth-wise convolutions do not perform
filtering across channels. Therefore, once the computation
of a channel is completed, the input activation of
the channel can be overwritten and used to store the
output activation of another channel, allowing activation
of depth-wise convolutions to be updated in-place
as shown in Figure 10. This method reduces the measured
memory usage by 1.6× as shown in Figure 8.
20
IEEE CIRCUITS AND SYSTEMS MAGAZINE
the per-block peak memory usage of MobileNetV2 [4] is
shown in Figure 11. The profiling is done in int8. There
is a clear pattern of imbalanced memory usage distribution.
The first five blocks have large peak memory, exceeding
the memory constraints of MCUs, while the remaining
13 blocks easily fit 256 KB memory constraints.
The third block has 8× larger memory usage than the
rest of the network, becoming the memory bottleneck.
There are similar patterns for other efficient network designs,
which is quite common across different CNN backbones,
even for models specialized for memory-limited
microcontrollers [8].
The phenomenon applies to most single-branch or
residual CNN designs due to the hierarchical structure1:
after each stage, the image resolution is down-sampled
by half, leading to 4× fewer pixels, while the channel
number increases only by 2× [5], [10], [132] or by an
even smaller ratio [4], [106], [133], resulting in a decreasing
activation size. Therefore, the memory bottleneck
tends to appear at the early stage of the network, after
which the peak memory usage is much smaller.
Breaking the Memory Bottleneck With PatchBased
Inference. TinEngine breaks the memory bottleneck
of the initial layers with patch-based inference
(Figure 12). Existing deep learning inference frameworks
(e.g., TensorFlow Lite Micro [90], TinyEngine [8],
microTVM [93], etc.) use a layer-by-layer execution. For
each convolutional layer, the inference library first allocates
the input and output activation buffer in SRAM,
and releases the input buffer after the whole layer computation
is finished. The patch-based inference runs
the initial memory-intensive stage in a patch-by-patch
1Some CNN designs have highly complicated branching structure (e.g.,
NASNet [83]), but they are generally less efficient for inference [6], [85],
[86]; thus, not widely used for edge computing.
THIRD QUARTER 2023

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023