IEEE Circuits and Systems Magazine - Q3 2023 - 27

Table 7.
Updating real quantized graphs (int8) with SGD is difficult: the transfer learning accuracy falls behind the floatingpoint
counterpart (fp32), even with adaptive learning rate optimizers like Adam [148] and LARS [149]. QAS helps to
bridge the accuracy gap without memory overhead (slightly higher). The numbers are for updating the last two blocks
of MCUNet-5FPS [8] model.
Precision
fp32
int8
Optimizer
SGD-M
SGD-M
Adam [148]
LARS [149]
SGDM+QAS
Accuracy
(%) MCUNet backbone: 23 M MACs, 0.48 M Param)
Cars
56.7
31.2
54.0
5.1
55.2
CF10
86.0
75.4
84.5
64.8
86.9
CF100
63.4
54.5
61.0
39.5
64.6
CUB
56.2
55.1
58.5
9.6
57.8
Flowers
88.8
84.5
87.2
28.8
89.1
Food
67.1
52.5
62.6
46.5
64.4
Avg
Pets VWW Acc.
79.5
81.0
80.1
39.1
88.7
85.4
86.5
85.0
80.9
89.3
73.3
64.9
71.8
39.8
73.5
data type (int8 versus int 32) and thus have scaling
factors of very different magnitude, leading to the zigzag
pattern. To solve the issue, quantization-aware scaling
(QAS) is proposed by compensating the gradient of the
quantized graph according to Equation (3):

GG GG
,
WW W bb Wx b
=⋅ =⋅ ⋅= ⋅
G
−− −−
ss ss22 22
(4)
where sX
−2 is the scaling factor for quantizing input
x (a scalar following [146], note that s = sW ⋅ sX in
Equation (1).  
WG/
Figure 16 (int8+scale). After scaling, the gradient ratios
match the floating-point counterpart. It also improves
transfer learning accuracy (Table 7), matching the accuracy
of the floating-point counterpart without incurring
memory overhead.
3) Experiment Results. The last two blocks in Table 7
show the fine-tuning results (simulating low-cost finetuning)
of MCUNet on various downstream datasets.
With momentum SGD, the training accuracy of the quantized
model (int 8) falls behind the floating-point counterpart
due to the difficulty in optimization. Adaptive
learning rate optimizers like Adam [148] can improve the
accuracy, but it is still lower than the fp32 fine-tuning
results. It also consumes three times more memory due
to second-order momentum, which is not desired in TinyML
settings. LARS [149] does not converge well on
most datasets despite extensive hyperparameter tuning
(of both the learning rate and the " trust coefficient " ). The
aggressive gradient scaling rule of LARS makes the training
unstable. The accuracy gap is closed when applying
QAS, achieving the same accuracy as floating-point training
with no extra memory cost. Figure 16 shows the training
curve of TinyTraining on the Cars dataset with and
without QAS. QAS effectively improves optimization.
THIRD QUARTER 2023
curve with QAS is plotted in
B. Memory-Efficient Sparse Update
Though QAS makes optimizing a quantized model possible,
updating the whole model (or even the last several
blocks) requires a large amount of memory, which
is not affordable for the TinyML setting. To address
this, sparsely updating the layers and the tensors is
proposed.
1) Sparse Layer/Tensor Update. Pruning techniques
prove to be quite successful for achieving sparsity and
reducing model size [58], [59], [60], [61], [62], [63]. Instead
of pruning weights for inference, the gradient during
backpropagation, and updating the model sparsely are
pruned. Given a tight memory budget, updates of the less
important parameters are skipped to reduce memory usage
and computation cost. When updating a linear layer
y = Wx + b (similar analysis applies to convolutions), the
gradient update is GW = f1(Gy,x) and Gb=f2(Gy), given the
output gradient Gy from the later layer Notice that updating
the biases does not require saving the intermediate
activation x, leading to a lighter memory footprint [52],5
while updating the weights is more memory-intensive
but also more expressive. For hardware like microcontrollers,
an extra copy is needed for the updated parameters
since the original ones are stored in read-only
FLASH [8]. Given the different natures of updating rules,
three aspects of the sparse update rule are considered
(Figure 17): (1) Bias update: how many layers should be
backpropagated to and update the biases (bias update
is cheap, the bias terms can be always updated if the
layer is backpropagated). (2) Sparse layer update: select
a subset of layers to update the corresponding weights.
(3) Sparse tensor update: further allow updating a subset
of weight channels to reduce the cost.
5If many layers are updated, the intermediate activation could consume
a large memory [109].
IEEE CIRCUITS AND SYSTEMS MAGAZINE
27
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
