IEEE Computational Intelligence Magazine - May 2022 - 43

as a form of variational inference, i.e., they learn the parameters
of an intermediate distribution to approximate the posterior.
All methods are summarized in Figure 9.
z +ij,
ii i
WM ^z h,
$
=
with zi
1) Bayes via Dropout
Dropout has initially been proposed as a regularization method
[94]. It works by applying multiplicative noise to the target
layer. The most commonly used type of noise is Bernoulli
noise, but other types such as the Gaussian noise for Gaussian
Dropout [94] might be used instead.
Dropout is usually turned off at evaluation time, but leaving
it on results in a distribution for the output predictions [95],
[96]. It turns out that this procedure, called Monte Carlo Dropout,
is in fact variational inference with a variational distribution
defined for each weight matrix as:
Algorithm 6 Bayes-by-backprop with parametric prior.
pp0;=
zz0;=
for i 0= to N do
Draw
izt f= ^
,, h
ff+ q ;^ h
,
h;
fq pD Dp ^
ff ;
izpi ih
T backprop=pp
^ =- ippz
^ h
loglog^
^ ^
T backprop ^ hff ;zz=
pp T f ;
zz T f ;
=-app
=-azz
end for
hh
hh
^ yx; ,;
Bernoull pi
diag
being the random activation coefficients and Mi
matrix of weights before dropout is applied. pi
^ ih ,
(41)
the
is the activation
probability for layer i and can be learned or set manually.
When used to train a BNN, dropout should not be seen as
a regularization method, as it is part of the variational posterior,
not the prior. This means that it should be coupled with a different
type of regularization [97], e.g.,
The equivalence between the objective function Ldropout
for training with dropout and 2
is defined as:
L
dropout
=+m i
i
N D
f^ht
yy
1// (42)
2
,,
i
and the ELBO, assuming a normal prior on the weights and
the distribution presented in Equation 41 as variational posterior,
has been demonstrated in [95]. The argument is similar to
the one presented in Section IV-C3.
MC-Dropout is a very convenient technique to perform
Bayesian deep learning. It is straightforward to implement and
requires little additional knowledge or modeling effort compared
to traditional methods. It often leads to a faster training
phase compared to other variational inference approaches. If a
model has been trained with dropout layers, which are quite
widespread in today's deep learning architectures, and an additional
form of regularization acting as prior, it can be used as a
BNN without any need to be retrained.
Benefits
MCMC (V.A)
Classic Methods
(HMC, NUTS)(§V-A)
SGLD and Derivates
(§V-E2a)
Warm Restarts (§V-E2a)
Variational Inference (V.B)
Bayes by Backprop (§V-C)
Monte Carlo-Dropout
(§V-E1)
Laplace Approximation
(§V-E2b)
Deep Ensembles
(§V-E2b)
Directly Samples the Posterior
State of the Art Samplers Limit
Autocorrelation Between Samples
Provide a Well Behaved
Markov Chain With Minibatches
Help a MCMC Method Explore
Different Modes of the Posterior
The Variational Distribution Is
Easy to Sample
Fit Any Parametric Distribution
as Posterior
Can Transform a Model
Using Dropout into a BNN
By Analyzing Standard SGD
get a BNN From a MAP
Help Focusing on Different
Modes of the Posterior
Limitations
Requires to Store a Very
Large Number of Samples
Do Not Scale Well
to Large Models
Focus on a Single Mode
of the Posterior
Requires a New Burn-In
Sequence for Each Restart
Is an Approximation
Noisy Gradient Descent
Lack Expressive Power
Focus on a Single Mode
Cannot Detect Local
Uncertainty if Used Alone
Use Cases
Small and Average
Models
Small and Critical
Models
Models With Larger
Datasets
Combined With
a MCMC Sampler
Large Scale Models
Large Scale Models
Dropout Based
Models
Unimodals
Large Scale Models
Multimodals Models
and Combined With
Other VI Methods
, weight penalization.
used
2
, weight regularization, which
FIGURE 9 Summary of the different inference approaches used to train a BNN with their benefits, limitations and use cases.
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 43
Can Be Combined
Can Be Combined
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
