IEEE Computational Intelligence Magazine - May 2022 - 40

Recent research has also shown that being only
approximately Bayesian is sufficient to achieve
a correctly calibrated model with uncertainty
estimates [27].
The acceptance probability p can be simplified if Q is chosen
to be symmetric, i.e., () ().QQnn
ii
p mincm.
1
=
f
f
()
()
l
in
i
() (, ), centered around the previous
ii; iiff= -+
l
Q nn or a uniform distribution
U
ii;
l
= i v
2
ll= ii The formula
;;
for the acceptance rate then becomes:
,
(34)
In this situation, the algorithm is simply called the Metropolis
method. Common choices for Q can be a normal distribution
() (, ),N
Q nn n
sample. To deal with non-symmetric proposal distributions, e.g.,
to accommodate a constraint in the model such as a bounded
domain, one has to take into account the correction term
imposed by the full Metropolis-Hasting algorithm.
The spread of ()Q n
ii;l
has to be tweaked. If it is too large,
the rejection rate will be too high. If it is too small, the samples
will be more autocorrelated. There is no general method to
tweak those parameters. However, a clever strategy to obtain the
new proposed sample il can reduce their impact. This is why
the Hamiltonian Monte-Carlo method has been proposed.
The Hamiltonian Monte Carlo algorithm (HMC)
[81] is another example of Metropolis-Hasting algorithms for
continuous distributions. It is designed with a clever scheme
to draw a new proposal il to ensure that as few samples as
possible are rejected and there is as few correlation as possible
between samples. In addition, the HMC's burn-in time is
extremely short compared to the standard Metropolis-Hasting
algorithm.
Most software packages for Bayesian statistics implement the
No-U-Turn sampler (NUTS for short) [82], which is an
improvement over the classic HMC algorithm allowing the
hyperparameters of the algorithm to be automatically tweaked
instead of manually setting them.
Algorithm 4 Metropolis-Hasting algorithm.
Draw
i + Initialprobabilitydistribution ;
while n 0= to N do
Draw
p mi ,;n
Q
=
ii ill^
^
Q
c1
n ;
+; nh ;
Q l;
ii
ii
n
^^
h ^
lh
iin 1+ = l ;
nn ;1=+
end if
end while
f
f
+ Bernoulli^ h
Drawkp ;
if k then
i
i
n
h
h
l
m
Dq Pq Hzz= # cm ldH .
z
KL () ()l log
<
H
Dq PKL
z <
()
()
PH D
qH
l
l;
one needs to compute () anyway. To overPH
D;
(35)
There is an apparent problem here, which is, to compute
(),
come this, a different, easily derived formula called the evidence
lower bound, or ELBO, serves as a loss:
# cm l=- <
dH
H
qH loglog
qH
z ()l
PH D(, )
z()l
l
(( )) (). (36)
PD Dq P
KL
Since lo (( ))
z <
z
g PD only depends on the prior, minimizing
Dq PKL () is equivalent to maximizing the ELBO.
The most popular method to optimize the ELBO is stochastic
variational inference (SVI) [85], which is in fact the stochastic
gradient descent method applied to variational inference. This
allows the algorithm to scale to the large datasets that are
encountered in modern machine learning, since the ELBO can
be computed on a single mini-batch at each iteration.
Convergence, when learning the posterior with SVI, will be
slow compared to the usual gradient descent. Moreover, most
implementations use a small number of samples to evaluate the
ELBO, often just one, before taking a gradient step. In other
words, the ELBO estimate will be noisy at each iteration.
In traditional machine learning and statistics, ()
qHz
is mostly
constructed from distributions in the exponential family, e.g.,
multivariate normal [86], Gamma and Dirichlet distributions.
The ELBO can then be dramatically simplified into components
[87]
leading
to a generalization of the well-known
expectation-maximization algorithm. To account for correlations
between the large number of parameters, certain approximations
are made. For instance, block diagonal [88] or low rank
plus diagonal [89] covariance matrices can be used to reduce
40 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
B. Variational Inference
MCMC algorithms are the best tools for
sampling from the exact posterior. However,
their lack of scalability has made them less
popular for BNNs, given the size of the
models under consideration. Variational inference
[77], which scales better than MCMC
algorithms, gained considerable popularity. Variational inference
is not an exact method. Rather than allowing sampling
from the exact posterior, the idea is to have a distribution
(),
called the variational distribution, parametrized by a
set of parameters z . The values of the parameters z are then
learned such that the variational distribution () is as close
as possible to the exact posterior ().PH D;
qHz
qHz
The measure of
closeness that is commonly used is the Kullback-Leibler divergence
(KL-divergence) [83]. It measures the differences
between probability distributions based on Shannon's information
theory [84]. The KL-divergence represents the average
number of additional bits required to encode a sample from P
using a code optimized for q. For Bayesian inference, it is
computed as:
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
