IEEE Circuits and Systems Magazine - Q3 2023 - 22

per-layer computation, the first convolutional layer
has large input and output activation size, leading to a
high peak memory. With spatial partial computation, it
allocates the buffer for the final output and computes
its values patch-by-patch. In this manner, it only needs
to store the activation from one patch instead of the
whole feature map.
Reducing Computation Overhead by Redistributing
the Receptive Field. The significant memory saving
comes at the cost of computation overhead. To maintain
the same output results as per-layer inference, the nonoverlapping
output patches correspond to overlapping
patches in the input image (the shadow area in Figure
12(b)). This is because convolutional filters with kernel
size >1 contribute to increasing receptive fields. The
computation overhead is related to the receptive field of
the patch-based initial stage. Consider the output of the
patch-based stage, the larger receptive field it has on the
input image, the larger resolution for each patch, leading
to a larger overlapping area and repeated computation.
There are some focusing on addressing the issue from
the hardware perspective [103]. However, since such
practices may not be general to all devices, TinyEngine
solves the problem from the network architecture side.
MCUNet proposes to redistribute the receptive field
(RF) of the CNN to reduce computation overhead. The
basic idea is: (1) reduce the receptive field of the patchbased
initial stage and (2) increase the receptive field of
the later stage. Reducing RF for the initial stage helps
to reduce the size of each input patch and repeated
computation. However, some tasks may have degraded
performance if the overall RF is smaller (e.g., detecting
large objects). Therefore, it further increases
the RF of the later stage to compensate for the performance
loss. A manually tuned example of MobileNetV2
is shown in Figure 13. After redistributing the
receptive field ( " MbV2-RD " ), the computation overhead
is negligible.
MCUNet automates the process with joint search (introduced
in the next section).
C. Co-Design: Joint Neural Architecture and
Inference Scheduling Search
1) Co-Design Loop
The optimization algorithms for model architectures and
inference engines are tightly coupled. For example, redistributing
the receptive field allows us to enjoy the benefit
of memory reduction at minimal computation/latency
overhead, which allows larger freedom when designing
the backbone architecture (e.g., we can now use a larger
input resolution). To explore such a large design space,
MCUNet jointly optimizes the neural architecture and the
inference scheduling in an automated manner. Given a certain
dataset and hardware constraints (SRAM limit, flash
limit, latency limit, etc.), our goal is to achieve the highest
accuracy while satisfying all the constraints. For the
model optimization, it uses NAS to find a good candidate
network architecture; for the scheduling optimization, it
optimizes the knobs like the patches number p and the
number of blocks n to perform patch-based inference,
and other knobs in TinyEngine [8].
There are some trade-offs during the co-design. For
example, given the same constraints, it can choose to
use a smaller model that fits per-layer execution (p=1,
no computation overhead), or a larger model and perpatch
inference (p > 1, with a small computation overhead).
Therefore, MCUNet puts both sides in the same
loop and uses evolutionary search to find the best set of
(k[], e[], d[], w[], r, p, n) satisfying constraints. Therefore,
the two dimensions are jointly searched in the same
loop with evolutionary search.
Figure 13. The redistributed MobileNetV2 (MbV2-RD) has reduced receptive field for the per-patch inference stage and
increased receptive field for the per-layer stage. The two networks have the same level of performance, but MbV2-RD has a
smaller overhead under patch-based inference. The mobile inverted block is denoted as MB {expansion ratio} {kernel
size}. The dashed border means stride = 2.
22
IEEE CIRCUITS AND SYSTEMS MAGAZINE
THIRD QUARTER 2023

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023