IEEE Computational Intelligence Magazine - May 2022 - 33

This approach is so common in ensemble learning that it is
sometimes called ensembling. To quantify uncertainty, the covariance
matrix can be computed as follows:
yx,D
// hh (7)-- <
;!H
= UUii^^hhi xy xy .
;;H -1 ii
1
i^^tt
When performing classification, the average model prediction
will give the relative probability of each class, which can
be considered a measure of uncertainty:
px1
;;H / U ().
t =
i Hi !
The final prediction is taken as the most likely class:
.
tt= argmax i !
ypp
i
ii
(8)
used for point estimate neural networks can be understood from
a Bayesian perspective as setting a prior; see Section IV-C3).
Finally, the Bayesian paradigm enables the analysis of
learning methods. A number of those methods initially not
presented as Bayesian can be implicitly understood as being
approximately Bayesian, e.g., regularization (Section IV-C3) or
ensembling (Section V-E2b). In fact, most of the BNNs used in
practice rely on methods that are approximately or implicitly
Bayesian (Section V-E) since the exact algorithms are computationally
too expensive. The Bayesian paradigm also provides a
systematic framework to design new learning and regularization
strategies, even for point estimate models.
BNNs have been used in many fields to quantify uncertain(9)
This
definition considers BNNs as discriminative models,
i.e., models that aim to reconstruct a target variable y given
observations x. This excludes generative models, although there
are examples of generative ANNs based on the Bayesian formalism,
e.g., Variational autoencoders [24]. Those are out of the
scope of this tutorial.
III. Advantages of Bayesian Methods for Deep Learning
One of the major critiques of Bayesian methods is that they
rely on prior knowledge. This is especially true in deep learning,
as deriving any insight about plausible parametrization for
a given model before training is very challenging. Thus, why
use Bayesian methods for deep learning? Discriminative models
implicitly represent the conditional probability (, ),yxp ; i and
Bayes' formula is an appropriate tool to invert conditional
probabilities, even if one has little insight about ()p i a priori.
While there are strong theoretical principles and schema upon
which this Bayes' formula can be based [25], this section focuses
on some practical benefits of using BNNs.
First, Bayesian methods provide a natural approach to
quantify uncertainty in deep learning since BNNs have better
calibration than classical neural networks [26]-[28], i.e.,
their uncertainty is more consistent with the observed errors.
They are less often overconfident or underconfident.
Second, a BNN allows distinguishing between the epistemic
uncertainty ()
pD;i
and the aleatoric uncertainty
p (, )yx; i [29]. This makes BNNs very data-efficient since they
can learn from a small dataset without overfitting [30]. At prediction
time, out-of-training distribution points will have high epistemic
uncertainty instead of blindly giving a wrong prediction.
Third, the no-free-lunch theorem for machine learning [31]
can be interpreted as stating that any supervised learning algorithm
includes some implicit prior. Bayesian methods, when
used correctly, will at least make the prior explicit. Integrating
prior knowledge into ANNs, which work as black boxes, is
difficult but not impossible. In Bayesian deep learning, priors are
often considered as soft constraints, analogous to regularization,
or data transformations such as data augmentation in traditional
deep learning; see Section IV-C. Most regularization methods
ty, e.g., in computer vision [32], network traffic monitoring
[33], aviation [34], civil engineering [35], [36], hydrology [37],
astronomy [38], electronics [39], and medicine [40]. BNNs are
useful in (1) active learning [41], [42] where an oracle (e.g., a
human annotator, a crowd, an expensive algorithm) can label
new points from an unlabeled dataset U. The model needs to
determine which points should be submitted to the oracle to
maximize its performance while minimizing the calls to the
oracle. BNNs are also useful in (2) online learning [43], where
the model is retrained multiple times as new data become
available. For active learning, data points in the training set
with high epistemic uncertainty are scheduled to be labeled
with higher priority; see Algorithm 2. In contrast, in online
Algorithm 1 Inference procedure for a BNN.
Define
pDh =
^i;
# ^ ll l
i
for i 0= to N do
Draw
i =U ii ^ h
yx ;
end for
return
Yiy Ni N00,;==H i ;! "" ,,;! hh
ii
,,
66
Algorithm 2 Active learning loop with a BNN.
while U Q! and
R
do
Draw H iii iN0+; !;hpD ,;
for x U! do
= "
;
^
R UU=yx, D
H -1
1 /
ifRR2yx yx
xx ;
max
end if
end for
DD x ;,maxxx
= , "
= , "
DD Oracle maxh, ;
UU\ xmax, ;
CC ;1=+
yy
= "
end while
^x
=
|,DDmax
|, then
6
h,
i di H^^ tt^iihh^
i -h
i
xy xy ;hR
yx Dmax
1
pD Dp d
pD Dp
^
yx
yx
;
;
iii pDh ;+;^
,
, ih
h
ii i
i
^
^
h
h
;
|, threshold and C MaxC1
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 33
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
