IEEE Circuits and Systems Magazine - Q3 2023 - 28

However, finding the right sparse update scheme
under a memory budget is challenging due to the large
combinational space. For MCUNet [8] model with 43
convolutional
layers and weight update ratios from
{0,1/8,1/4,1/2,1}, the combination is about 1030, making
exhaustive search impossible.
2) Automated Selection With Contribution Analysis.
Contribution analysis is proposed to automatically derive
the sparse update scheme by counting the contribution
of each parameter (weight/bias) to the downstream
accuracy. Given a convolutional neural network with l
layers, the accuracy improvement is measured from (1)
biases: the improvement of updating last k biases bl, bl−1,
..., bl−k+1 (bias-only update) compared to only updating
the classifier, defined as Δaccb[:k] and (2) weights: the improvement
of updating the weight of one extra layer Wi
(with a channel update ratio r) compared to bias-only
update, defined as ΔaccWi,r. An example of the contribution
analysis can be found in Figure 18 left (MCUNet
on Cars [150] dataset. After finding Δaccb[:k] and ΔaccWi
(1≤k,i≤l), an optimization problem is solved to find:
k ir ∆∆max accb[] ∑ accWir
∗∗ ∗
,,
=+

,,


oroyconstraint,
k,,
ir
k ir 
:k
∈∈,ir
ir
s.t. Mem ()≤
,




(5)
where i is a collection of layer indices whose weights are
updated, and r is the corresponding update ratios (1/8,
1/4, 1/2, 1). Intuitively, by solving this optimization problem,
the combination of (#layers for bias update is found,
the subset of weights to update), such that the total
contribution is maximized while the memory overhead
does not exceed the constraint. The problem can be
efficiently solved with the evolutionary search. Sparse
update assumes that the accuracy contribution of each
tensor (Δacc) can be summed up. Such an approximation
proves to be quite effective in our experiments.
3) Sparse Update Obtains Better Accuracy at Lower
Memory. The performance of our searched sparse update
schemes is compared to two baseline methods:
fine-tuning only the biases of the last k layers and finetuning
the weights and biases of the last k layers. For
each configuration, the average accuracy is measured
on eight downstream datasets, and the extra memory
usage is calculated analytically. Figure 18 compares
the results with a simple baseline of fine-tuning only
the classifier. The accuracy of classifier-only update is
low due to the limited learning capacity. Updating only
the classifier is not enough; the backbone also needs
updates. Bias-only update outperforms classifier-only
update, but the accuracy quickly plateaus and does not
Figure 17. Different learning schemes on ProxylessNAS-Mobile [85]. Full update (a) consumes a lot of memory thus cannot fit
TinyML. Efficient learning methods like (b) last-only/(c) bias-only save the memory but cannot match the baseline performance.
Sparse update (d) only performs partial back-propagation, leading to less memory usage and computation with comparable
accuracy on downstream tasks.
28
IEEE CIRCUITS AND SYSTEMS MAGAZINE
THIRD QUARTER 2023
IEEE Circuits and Systems Magazine - Q3 2023

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q3 2023
