IEEE Circuits and Systems Magazine - Q3 2023 - 17

only fine-tuning the biases, which allows intermediate
activations to be discarded during backward propagation,
reducing peak memory usage [52]. TinyOL trains
only the weights of the final layer, allowing for weight
training while keeping the activation small enough
to fit on small devices [53]. This enables incremental
on-device streaming of data for training. However, finetuning
only the biases or the last layer may not provide
sufficient precision. To train more layers on devices
with limited memory, POET (private optimal energy
training) [54] introduces two techniques: rematerialization,
which frees up activations early at the cost of
recomputation, and paging, which allows activations
to be transferred to secondary storage. POET uses
an integer linear program to find the energy-optimal
schedule for on-device training. To further reduce the
memory required to store trained weights, MiniLearn
applies quantization and dequantization techniques to
store the weights and intermediate output in integer
precision and dequantizes them to floating-point precision
during training [55]. When deployed on tiny devices,
deep learning models are often quantized to reduce
the memory usage of parameters and activations.
However, even after quantization, the parameters may
still be too large to fit in the limited hardware resources,
preventing full back-propagation. To address these
challenges, MCUNetV3 proposes an algorithm-system
co-design approach [56]. The algorithm part includes
quantization-aware scaling (QAS) and the sparse update.
QAS calibrates the gradient scales and stabilizes
8-bit quantized training, while the sparse update skips
the gradient computation of less important layers and
sub-tensors. The system part includes the tiny training
engine (TTE), which has been developed to support
both QAS and the sparse update, enabling on-device
learning on microcontrollers with limited memory,
such as those with 256 KB or even less.
III. Tiny Inference
In this section, we discuss our recent work, MCUNet
family [8], [9], a system-algorithm co-design framework
that jointly optimizes the NN architecture (TinyNAS)
and the inference scheduling (TinyEngine) in the same
loop (Figure 4). Compared to traditional methods that
either (a) optimize the neural network using neural architecture
search based on a given deep learning library
(e.g., TensorFlow, PyTorch) [85], [86], [87] or (b) tune the
library to maximize the inference speed for a given network
[93], [99], MCUNet can better utilize the resources
by system-algorithm co-design, enabling a better performance
on microcontrollers. The design space of the
inference part is listed in Figure 3 (left).
A. TinyNAS: Automated Tiny Model Design
TinyNAS is a two-stage neural architecture search method
that first optimizes the search space to fit the tiny
and diverse resource constraints, and then performs
neural architecture search within the optimized space.
By optimizing the search space, it significantly improves
the accuracy of the final model.
1) Automated Search Space Optimization
TinyNAS proposes to optimize the search space automatically
at low cost by analyzing the computation distribution
of the satisfying models. To fit the tiny and diverse
resource constraints of different microcontrollers,
TinyNAS scales the input resolution and the width multiplier
of the mobile search space [86]. It chooses from an
input resolution spanning R = {48, 64, 80, ..., 192, 208,
224} and a width multiplier W = {0.2, 0.3, 0.4, ..., 1.0} to
cover a wide spectrum of resource constraints. This
leads to 12 × 9 = 108 possible search space configurations
S = W × R. Each search space configuration contains
3.3 × 1025 possible sub-networks. The goal is to find
the best search space configuration S* that contains the
Figure 3. Techniques specifically designed for tiny devices. In order to fully leverage the limited available resources, we need to
take careful consideration of both the system and the algorithm. The co-design approach not only enables practical AI applications
on a wide range of IoT platforms (inference), but also allows AI to continuously learn over time, adapting to a world that is
changing fast (training).
THIRD QUARTER 2023
IEEE CIRCUITS AND SYSTEMS MAGAZINE
17

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023