IEEE Computational Intelligence Magazine - November 2022 - 48

overall loss (Theorem 6 in [22]), thus leading to effective
training.
Almost all existing theoretical optimality results require
that the network consists ofpurely linear layers (possibly linked
by elementwise nonlinearities).
c) Advantages: As a weakly modular method, Target Propagation
does not require end-to-end backward pass and updates
each module individually after each forward pass is done. When
computing gradients for the entire model using the chain rule
becomes expensive, Target Propagation may therefore help
save computations since it only needs module-wise gradients.
This also enables training models that have non-differentiable
operations, as demonstrated in [14]. These advantages are of
course shared by the other two families ofmethods.
Note that if and how much computation can be saved
highly depends on the actual use case. To be specific, while
Target Propagation does not need full backward pass, it does
require nontrivial computations for calculating the hidden targets
that involve training and evaluating the auxiliary models.
Depending on how complex these models are, this extra
workload may overweigh the saving of eliminating full backward
pass. Further, some Target Propagation variants require
stashing forward pass results for evaluating hidden targets,
meaning that they may not have any advantage over E2EBP
in terms ofmemory footprint either.
d) Current Limitations and Future Work:
❏ The auxiliary models require extra human (architecture
selection, hyperparameter tuning, etc.) and machine
resources.
❏ Target Propagation methods have not been shown to yield
strong performance on more challenging datasets such as
CIFAR-10 and ImageNet or on more competitive networks
[22], [63].
❏ Similar to Proxy Objective methods, optimality results for
more general settings, in particular, broader network architecture
families, are lacking.
2) Synthetic Gradients
Synthetic Gradients methods approximate local gradients and
use those in place of real gradients produced by end-to-end
backward pass for training. Specifically, Synthetic Gradients
methods assume that the network weights are updated using a
gradient-based optimization algorithm such as stochastic gradient
descent. Then these methods approximate local gradients
with auxiliary models. These auxiliary models are typically
implemented with fully-connected networks, and are trained
to regress to a module's gradients (gradients of the overall
objective function with respect to the module's activations)
when given its activations. By leveraging these local gradient
models, Synthetic Gradients methods reduce the frequency in
which end-to-end backward passes are needed by using the
synthesized gradients in place of real gradients. End-to-end
backward passes are only performed occasionally to collect real
48 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2022
gradients for training the local gradient models. Re-writing
Lðf; u1; u2; ðx; yÞÞ as Hðfðx; u1; u2Þ; y; u1; u2Þ for some H,
these methods can be abstracted as Algorithm 3.
It is possible to reduce the frequency that the forward pass
is needed as well by approximating the forward pass signals
with auxiliary synthetic input models that predict inputs to
modules given data.
Synthetic Gradients methods do not allow truly weakly
modular training since occasional end-to-end backward (or
forward) passes are needed to learn the synthetic gradient (or
input) models. They can only be used to accelerate end-toend
training and enable parallelized optimizations.
a) Instantiations:
❏ [16], [17]: Proposed the original instantiation, which is
fully described above.
❏ To remove the need for occasional end-to-end backward
pass, [18] proposed a method to obtain target signals for
training the synthetic gradient models using only local
information. However, this method necessitates the use of
stochastic networks. And the performance reported in the
paper is underwhelming compared to either [16] or
Algorithm 3. An Optimization Step in Synthetic Gradients
Require: An auxiliary synthetic gradient model s2ð; c2Þ, training
data ðxi; yiÞ, step size h > 0
1: begin
2: End-to-end forward pass
a1 u1ðÞ f1 xi; u1ðÞ; a2 u1; u2ðÞ f2 a1 u1ðÞ; u2ðÞ (16)
3: Update output module
u2 u2 h
@Hf xi; u1; uðÞ; yi; u1; uðÞ
@u
(17)
u¼u2
4: Obtain synthetic gradients for input module
^d1 u1; c2ðÞ s2 a1 u1ðÞ; c2ðÞ (18)
5: Update input module
u1 u1 h^d1 u1; c2ðÞ
6:
7:
@a1ðuÞ
@u
(19)
u¼u1
if update synthetic gradient model then
End-to-end backward pass, obtain true gradients
d1
@Hf2 u; u2ðÞ; yi; u1; u2ðÞ
@u
8:
Update c2 to minimize
d1^d1 u1; c2ðÞ
9: end if
10: end
2
2
u¼a1 u1ðÞ
(20)
(21)
IEEE Computational Intelligence Magazine - November 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2022
