IEEE Circuits and Systems Magazine - Q3 2023 - 29

Figure 18. Contribution analysis of updating biases and weights. For updating the weight of a specific layer, the later layers
appear to be more important; the first point-wise conv (pw1) in an inverted bottleneck block [4] appears to be more important;
and the gains are bigger with more channels updated (left). Sparse update can achieve higher transfer learning accuracy using
4.5−7.5× smaller extra memory (analytic) compared to updating the last k layers (right). For classifier-only update, the accuracy
is low due to limited capacity. Bias-only update can achieve a higher accuracy but plateaus soon. (a) Contribution of a certain
weight laccWi,r
(b) MbV2-w0.35 accuracy comparison.
Figure 19. The workflow of our tiny training engine (TTE). (a) and (b) Our engine traces the forward graph for a given model
and derives the corresponding backward graph at compile time. The red cycles denote the gradient descent operators. (c) To
reduce memory requirements, nodes related with frozen weights (colored in light blue) are pruned from backward computation.
(d) To minimize memory footprint, the gradient descent operators are re-ordered to be interlaced with backward computations
(colored in yellow). (e) TTE compiles forward and backward graphs using code generation and deploys training on tiny IoT
devices (best viewed in colors).
improve even when more biases are tuned. For updating
the last k layers, the accuracy generally improves
as more layers are tuned; however, it has a very large
memory footprint. For example, updating the last two
blocks of MCUNet leads to an extra memory usage exceeding
256 KB, making it infeasible for IoT devices/microcontrollers.
Our sparse update scheme can achieve
higher downstream accuracy at a much lower memory
cost. Compared to updating the last k layers, the sparse
update can achieve higher downstream accuracy with
4.5-7.5 times smaller memory overhead The highest accuracy
is achievable by updating the last k layers6 as
the baseline upper bound (denoted as " upper bound " ).
Interestingly, our sparse update achieves a better downstream
accuracy compared to the baseline best statistics.
The sparse update scheme alleviates over-fitting or
makes momentum-free optimization easier.
6Note that fine-tuning the entire model does not always lead to the best
accuracy. The best k on Cars dataset is obtained via grid search: k = 36
for MobileNetV2, 39 for ProxylessNAS, 12 for MCUNet, and apply it to
all datasets.
THIRD QUARTER 2023
C. Tiny Training Engine (TTE)
The theoretical saving from real quantized training and
sparse update does not translate to measured memory
saving in existing deep learning frameworks, due to
the redundant runtime and the lack of graph pruning.
MCUNetV3 co-designed an efficient training system,
tiny training engine (TTE), to transform the above algorithms
into slim binary codes (Figure 19).
1) Compile-Time Differentiation and Code Generation.
TTE offloads the auto-differentiation from the runtime to
the compile-time, generating a static backward graph that
can be pruned and optimized (see below) to reduce the
memory and computation. TTE is based on code generation:
it compiles the optimized graphs to executable binaries
on the target hardware, which minimizes the runtime
library size and removes the need for host languages like
Python (typically uses Megabytes of memory).
2) Backward Graph Pruning for Sparse Update. TTE
prune the redundant nodes in the backward graph
IEEE CIRCUITS AND SYSTEMS MAGAZINE
29

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023