IEEE Computational Intelligence Magazine - May 2022 - 41

the number of variational parameters z from ()O n2
to (),O n
where n is the number of model parameters i . Appendix A
gives more details on how these simplifications are implemented
in practice.
Appendix A
Implementing parameter-efficient normal distributions
for variational inference
Given a random vector f in which each independent and
identically distributed component follows a standard normal
distribution
mal distribution
N( ,)n R using the following formulas:
if ,nR=+
where R is a matrix such that
RR .R=
<
n entries, RR is the Cholesky decomposition of R . As
such, R is a lower triangular matrix and ()O n2
i , it is often more convenient to learn
<
variational
parameters are required to learn the exact covariance matrix.
This becomes exceedingly computationally expensive rather
quickly when n becomes large.
A straightforward simplification is to only consider a diagonal
approximation of R . This can be enforced by learning only the
diagonal coefficients of
R , meaning only ()O n variational
parameters are required (Figure A1a). This approach can be
extended to learn more correlation coefficients by learning a block
diagonal [88] covariance matrix, which can be done by learning
the corresponding lower triangular entries in R (Figure A1b). If
the maximal size of the nonzero blocks is fixed to be w, ()O wn$
variational parameters are required. The major drawback of this
model is that the index of two given parameters determines
whether their covariance can be learned by the variational distribution.
This is not always ideal, as it is hard to predict which
parameters will be the most correlated and need to be positioned
close to one another. An alternative is to learn a diagonal plus low
rank approximation of R [89]. This is done by sampling a vector
f with nr+ (instead of n) components. R is then defined as:
R DL= 6 ,,
@
where D is a diagonal matrix of size nn#
O( )rn $
(56)
and L is a lower triangular
matrix of size nr# (Figure A1c). This means that the model
has more flexibility to learn the correlation between all the components
of i while only requiring
variational parameters.
Algorithm 5 Bayes-by-backprop algorithm.
zz0;=
for i 0= to N do
Draw
(a)
(b)
(c)
FIGURE A1 Nonzero entries in R when learning a diagonal (a),
block diagonal (b) or diagonal plus low rank (c) approximation of R
izt f= ^
h
ff+ q ;^ h
,
h;
fq pD Dp^ ,, ;hh
^iz ii i=-z
T backprop ^ hff ;zz=
zz T f ;a=- z
loglog^ ^ yx;
^ ^
end for
hh
h
This is tedious to work with. Instead, to estimate the gradient
of the ELBO, Blundell et al. [91] proposed to use the fact that if
()
z
then for a differentiable function (, ),f iz
2z z
z ii zi=+ .
2z
() (, )( )
f
(, )( ,)
2i
iz
2
2
z
i
2 iz
(38)
A proof is provided in [91]. We also provide in Appendix B an
alternative proof to give more details on when we can assume
()
qd () .qdii ff=
z
A sufficient condition is for (, ) to be
t zf
invertible with respect to f and the distributions ()q f and
q ()iz
to not be degenerated.
For the case where the weights are treated as stochastic variables,
and thus the hypothesis H, the training loop can be
implemented as described in Algorithm 5.
The objective function f corresponds to an estimate of the
ELBO from a single sample. This means that the gradient
2##ffqf dqll l cmd
2ff
qd () ,qdii ff=
we have:
N( ,),01 one can obtain a sample i from a nor(55)
When
a variational
inference algorithm needs to learn the covariance matrix
of
R . Assuming i has
C. Bayes by Backpropagation
Variational inference offers a good mathematical tool for
Bayesian inference, but it needs to be adapted to deep learning.
The main problem is that stochasticity stops backpropagation
from functioning at the internal nodes of a network [46]. Different
solutions have been proposed to mitigate this problem,
including probabilistic backpropagation [90] or Bayes-by-backprop
[91]. The latter may appear more familiar to deep learning
practitioners. We will thus focus on Bayes-by-backprop in this
tutorial. Bayes-by-backprop is indeed a practical implementation
of SVI combined with a reparametrization trick [92] to
ensure backpropagation works as usual.
The idea is to use a random variable
ff+ q () as a nonvariational
source of noise. i is not sampled directly but obtained
via a deterministic transformation (, )
q iz
t zf
such that izt (, )f=
follows (). f is sampled and thus changes at each iteration
but can still be considered a constant with regard to other variables.
All other transformations being non-stochastic, backpropagation
works as usual for the variational parameters
z ,
meaning the training loop can be implemented analogous to
the training loop of a non-stochastic neural network; see Algorithm
5. The general formula for the ELBO becomes:
# cm (37)
Pt f z D
qt f z
f
z (( ,)) log
(( ,),)
z(( ,))
qt f z
De tdt d ff(( ,)).
f
z
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 41
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
