IEEE Circuits and Systems Magazine - Q3 2023 - 30

before compiling it to binary codes. For sparse layer update,
TTE prune away the gradient nodes of the frozen
weights, only keeping the nodes for bias update. Afterward,
TTE traverses the graph to find unused intermediate
nodes due to pruning (e.g., saved input activation)
and apply dead-code elimination (DCE) to remove the
redundancy. For sparse tensor update, TTE introduces a
sub-operator slicing mechanism to split a layer's weights
into trainable and frozen parts; the backward graph of
the frozen subset is removed. TTE's compilation translates
the sparse update algorithm into the measured
memory saving, reducing the training memory by 7−9×
without losing accuracy (Figure 21(a)).
3) Operator Reordering and Graph Optimization. The
execution order of different operations affects the life
cycle of tensors and the overall memory footprint. This
has been well-studied for inference [44], [101] but not for
training due to the extra complexity. Traditional training
frameworks usually derive the gradients of all the
trainable parameters before applying the update. Such
a practice leads to significant memory waste for storing
the gradients. By reordering operators, the gradient
update to a specific tensor can immediately be applied
(in-place update) before back-propagating to earlier layers,
so that the gradient can be released. As such, TTE
trace the dependency of all tensors (weights, gradients,
activation) and reorder the operators, so that some operators
can be fused to reduce memory footprint (by
2.4-3.2 ×, Figure 21(a)). Figure 20 provides an example to
reflect the memory saving from reordering.
4) Memory Saving and Faster Training. Figure 21(a)
shows the training memory of three models on STM32F746
MCU to compare the memory saving from TTE. The
sparse update effectively reduces peak memory by 7−9×
Figure 20. Memory footprint reduction by operator reordering. With operator reordering, TTE can apply in-place gradient update
and perform operator fusion to avoid large intermediate tensors to reduce memory footprint. We profiled MobileNetV2-w0.35 in
this figure (same as Figure 15). (a) Vanilla backward graph. (b) Optimized backward graph.
Figure 21. Measured peak memory and latency: (a) sparse update with TTE graph optimization can reduce the measured peak
memory by 20-21 × for different models, making training feasible on tiny edge devices. (b) Graph optimization consistently
reduces the peak memory for different sparse update schemes (denoted by different average transfer learning accuracies).
(c) Sparse update with TTE operators achieves 23-25× faster training speed compared to the full update with TF-Lite Micro
operators, leading to less energy usage. Note: for sparse update, we choose the config that achieves the same accuracy as
full update.
30
IEEE CIRCUITS AND SYSTEMS MAGAZINE
THIRD QUARTER 2023

IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023