IEEE Computational Intelligence Magazine - May 2022 - 42

estimate will be noisy. The convergence graph will also be
much more noisy than in the case of classic backpropagation
(Figure 8). To obtain a better estimate of the convergence, one
can average the loss over multiple epochs.
Since algorithm 5 is very similar to the classical training
loop for point estimate deep learning, most techniques used
Appendix B
A Proof of Equation 38
Let us assume that we have a probability space (, ),F, PX
X is a set of outcomes, F is a v -algebra of X representing poswhere
sible
events and P is a measure defined on F and which assigns
a value of 1 to
X , representing the probability of an event. In
addition, assume that we have a probability distribution ()q iz
for a given random variable
t zf
i , a probability distribution ()q f for
t zf
t zf
a given random variable f and a functional relation (, ) such
that (, ) is distributed according to ()q iz
bijection with respect to f . Thus, we have:
P 11
tE
with
tE tE
Ee
ff~~
ff
^
f
-1^^
^FFee
^
-1^
,, :,
,
dd
zz d
f
izh
t zf
iz1 tE ,.
- ^ ^
!!E
izh
tE,
^
z i i =
f
tE,.
h
h
h
h
=
=
=
=
'
'
ee E
ee tE
d/
d/
F
F
3
iz3
()
() (, )
e
Since (, ) is a bijection with respect to f , we have
hh This implies:
qd qd E^^ ^hh hF .
ff 6 ! f
f 1 E =
- ^ h
## (58)
which in turn implies:
qd qdi i ff= ^ h
z ^
h
Now, given a differentiable function f zi^
,
izfhhh
!!,
t^^^
XX
^
fq df tq d
zi ii zz^ ff f ,, ,
h
z^
h
=
^
hh
^ h
ff
,
(59)
for non-degenerated probability distributions q iz ^ h and q .f^ h
h we have:
## (60)
which implies Equation (38).
In general, directly finding pt is an intractable problem. However,
when using variational inference, the ELBO is the log likelihood
of the data minus the KL-divergence of q iz
(Eq. 36):
lo |.g PD
^^ p =+ z
hh
ELBO Dq PKL
^
<
h
"
"
" ^ :: ,
h
h
,
,
,
(( (, ))) (( )) 6 !
P EE
iz ff=-- ( ),F
(57)
for optimization in deep learning are straightforward to use
for Bayes-by-backprop. For example, it is perfectly fine to
use the ADAM optimizer [93] instead of the stochastic gradient
descent.
Note also that, if Bayes-by-backprop is presented for BNNs
with stochastic weights, adapting it for BNNs with stochastic
activations is straightforward. In that case, the activations l represent
the hypothesis H and the weights i are part of the variational
parameters
z .
and (, ) is a
D. Learning the Prior
Learning the prior and the posterior afterwards is possible. This
is meaningful if most aspects of the prior can be set using prior
knowledge, and only a limited set of free parameters of the
prior are learned before obtaining the posterior. In standard
Bayesian statistics, this is known as empirical Bayes. This is
usually a valid approximation when the dimensions of the prior
parameters being learned are significantly smaller than the
dimensions of the model parameters.
Given a parametrized prior distribution
pH ,^h maximizp
ing
the likelihood of the data is a good method to learn the
parameters :p
t
argmax
argmax
p
p
pp=
=
PD;
^
H
h
# pD Hp HdHpp;
^ ll l .
h
^
h
(39)
^h and prior
(40)
This property means that maximizing the ELBO, now a function
of both p and
is from a general family of probabiliz
, is equivalent to maximizing a lower
bound on the log likelihood of the data. This lower bound
becomes tighter when qz
ty distributions with more flexibility to fit the exact posterior
PD | .
^hi
The Bayes-by-backprop algorithm presented in Section
V-C needs only to be slightly modified to include the
additional parameters in the training loop; see Algorithm 6.
100
10-1
2,000 4,000
6,000 8,000 10,000
Epoch
FIGURE 8 Typical training curve for Bayes-by-backprop.
42 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
Loss
Average Loss
E. Inference Algorithms Adapted for Deep Learning
We presented thus far the fundamental theory to design and
train BNNs. However, the aforementioned methods are still
not easily applicable to most large scale architectures currently
used in deep learning. Recent research has also shown that
being only approximately Bayesian is sufficient to achieve a
correctly calibrated model with uncertainty estimates [27]. This
section presents how inference algorithms were adapted for
deep learning, resulting in more efficient methods. Specific
inference methods can still be classified as MCMC algorithms,
i.e., they generate a sequence of samples from the posterior, or
-ELBO/|D| [Log Scale]
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
