IEEE Computational Intelligence Magazine - May 2022 - 32

... the Bayesian paradigm enables the analysis of
learning methods. A number of those methods
initially not presented as Bayesian can be implicitly
understood as being approximate Bayesian ...
a stochastic neural network architecture is to obtain a better idea
of the uncertainty associated with the underlying processes. This is
accomplished by comparing the predictions of multiple sampled
model parametrizations
i . If the different models agree, then the
uncertainty is low. If they disagree, then the uncertainty is high.
This process can be summarized as follows:
(),
yx
ii+ p
=+Ui
() ,
e
(3)
where e represents random noise to account for the fact that
the function U is only an approximation. A BNN can then be
defined as any stochastic artificial neural network trained using
Bayesian inference [21].
To design a BNN, the first step is the choice of a deep neural
network architecture, i.e., a functional model. Then, one has
to choose a stochastic model, i.e., a prior distribution over the
possible model parametrization ()p i and a prior confidence in
the predictive power of the model (, )yxp ; i (Figure 2a). The
model parametrization can be considered to be the hypothesis H
and the training set is the data D. The choice of a BNN's stochastic
model is somehow equivalent to the choice of a loss function
when training a point estimate neural network; see Section
IV-C3. In the rest of this paper, we will denote the model parameters
by ,i the training set by D, the training inputs by
the training labels by D .y By applying Bayes' theorem, and
D ,x
and
enforcing independence between the model parameters and the
input, the Bayesian posterior can be written as:
pD = ();
i
#
i
pD Dp d
pD Dp
(, )( )
(, )( )
yx
yx
;
;
ii i
ii
ll l
?;
pD Dp
(, )( ).
yx ii (4)
two broad approaches have been introduced: (1) Markov chain
Monte Carlo and (2) variational inference. These are presented
in more details in Section V.
When using a BNN for prediction, the probability
pD;
i
distribution (, )yx
[12], called the marginal and which
quantifies the model's uncertainty on its prediction, is of
particular interest. Given (),( ,)yxpD pD
puted as:
i;; can be compD
yx ll l
(, )( ,) () .
yx
;; ;ii ipp Dd
= #
i
pD;
In practice, (, )yx
(5)
is sampled indirectly using Equation (3).
The final prediction can be summarized by statistics computed
using a Monte Carlo approach (Figure 2c). A large set of
weights ii is sampled from the posterior and used to compute
a series of possible outputs ,yi
as shown in Algorithm 1, which
corresponds to samples from the marginal.
In Algorithm 1, Y is a set of samples from (, )yx
a collection of samples from ().pD;i
pD;
and H
Usually, aggregates are
computed on those samples to summarize the uncertainty of
the BNN and obtain an estimator for the output y. This estimator
is denoted by y.t
When performing regression, the procedure that is usually
used to summarize the predictions of a BNN is model averaging
[23]:
yx1
;;H / U ().
t =
i Hi !
ii
(6)
#pD Dp d
(, )( )
The Bayesian posterior for complex models
such as artificial neural networks is a high
dimensional and highly non-convex probability
distribution [22]. This complexity makes
computing and sampling it using standard
methods an intractable problem, especially
because computing the evidence
yx; ii ill l is difficult. To address this problem,
(a)
(b)
(c)
FIGURE 3 (a) Point estimate neural network, (b) stochastic neural network with a probability distribution for the activations, and (c) stochastic
neural network with a probability distribution over the weights.
32 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
