IEEE Computational Intelligence Magazine - May 2022 - 45

interpreted as approximating the posterior with a distribution
parametrized as multiple Dirac deltas, i.e.,
q^^ ,hh/
i
zi
iz
with the i
ai
iiad=
!
i
ii
(46)
being positive constants such that their sum is
equal to one. This approach can be seen as a form of variational
inference. Note however that, for a variational distribution
containing Dirac deltas, computing the ELBO in a sense that is
meaningful for traditional optimization is impossible.
VI. Simplifying Bayesian Neural Networks
After training a BNN, one has to use Monte Carlo at evaluation
time to estimate uncertainty. This is a major drawback of
BNNs. For MCMC-based methods, storing a large set of
parametrizations H is also not practical. This section presents
mitigation strategies reported in the literature.
A. Bayesian Inference on the (N-)Last Layer(s) Only
The architecture of deep neural networks makes it quite
redundant to account for uncertainty for a large number of
successive layers. Instead, recent research aims to use only a few
stochastic layers, usually positioned at the end of the networks
[105], [106]; see Figure 11. With only a few stochastic layers,
training and evaluation can be drastically sped up while still
obtaining meaningful results from a Bayesian perspective. This
approach can be seen as learning a point estimate transformation
followed by a shallow BNN.
Training a BNN with some non-stochastic layers is similar
to learning the parameters for the prior presented in Section
V-D. The weights of the non-Bayesian layers should be considered
as both prior and variational-posterior parameters.
B. Bayesian Teachers
Using a BNN as a teacher is an idea derived from an approach
used in Bayesian modeling [107]. The approach is to train a
non-stochastic ANN to predict the marginal probability
yx ,
can be much larger than D. This helps the student network
retain the calibration and uncertainty from the teacher.
Menon et al. [110] observed that, for classification problems,
simply using the class probabilities output by a BNN teacher
rather than one-hot labels helps the student to retain calibration
and uncertainty from the teacher.
A Bayesian teacher can also be used to compress a large set
of samples generated using MCMC [111]. Instead of storing ,H
a generative model G (e.g., a GAN in [111]) is trained against
the MCMC samples to generate the coefficients ii at evaluation
time. This approach is similar to variational inference, with
G representing a parametric distribution, but the proposed
MCMC Methods
Variational Inference
Deep Ensembles With SGD
FIGURE 10 Different techniques for sampling the posterior. MCMC
algorithms sample the true posterior but successive samples might
be correlated, Variational Inference uses a parametric distribution
that can suffer from mode collapse while deep ensembles focus on
the modes of the distribution.
Algorithm 8 Deep ensembles.
for i 0= to R do
Draw
e e=
for j 0= to N do
t
pD;^h using a BNN as a teacher [108]. This is related to
the idea of knowledge distillation [109], [110] where possibly
several pre-trained knowledge sources can be used to train a
more functional system.
To do so, the KL-divergence between a parametric distribu~
;^h where ~ are the coefficients of the student netyx
,
tion q yx ,
work, and pD;^h is minimized:
x
t
~ = argmin ,.yxDp Dq
KL^^^ ;;<
h
~
As this is intractable, Korattikara et al. [108] proposed a Monte
Carlo approximation:
t
~
=- ;
!
argmin
~
1 / E ^p yx, ii h log yxq^ ^
H ii H
6
~
;
hh@ .
(48)
Here, ~t can be estimated using a training dataset Dl that
contains only the features x. During training, the probability
p yx ,; i^h of the labels is given by the teacher BNN. Thus, Dl
D
y
FIGURE 11 PGM and BNN corresponding to a last-layer model.
l
θ
~ yxhh
(47)
ξ
ii Ta=- i i
end for
ii
end for
fpDD ijhh
T backprop=ii
f ;
ijh
^ii;glog^ ^=+ i,, p ,ijhh
ff ;
lo ,;^^ yx
^ h
i + Initialprobabilitydistribution ;
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 45
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
