IEEE Computational Intelligence Magazine - May 2022 - 31

It is intended to motivate and help researchers and students to
use BNNs in measuring uncertainty for problems in their
respective fields of study and research, helping them relate
their existing knowledge in deep learning to the relevant
Bayesian methods.
The remaining parts of this paper are organized as follows.
Section II introduces the concept of a BNN. Section III presents
the motivations for BNNs as well as their applications.
Section IV explains how to design the stochastic model associated
with a BNN. Section V explores the most important algorithms
used for Bayesian inference and how they were adapted
for deep learning. Section VI reviews BNN simplification
methods. Section VII presents the methods used to evaluate the
performance of a BNN. Finally, Section VIII concludes the
paper. The supplementary material contains a gallery of practical
examples illustrating the theoretical concepts presented in
Sections II, IV and V of the main paper. Each example source
code is also available online on GitHub to provide implementation
examples of the most important algorithms to work
with BNNs.
II. What is a Bayesian Neural Network?
A BNN is defined slightly differently across the literature, but a
commonly agreed definition is that a BNN is a stochastic artificial
neural network trained using Bayesian inference.
The goal of artificial neural networks (ANNs) is to represent
an arbitrary function
yxU=
l ,0
a
in
i 11f=- and one output layer .ln
succession of hidden layers
(Here, n 1+ is the
Here, (, )Wbi =
are the parameters of the network, where W
are the weights of the network connections and b the biases. A
given ANN architecture represents a set of functions isomorphic
to the set of possible parameters
i . Deep learning is the
process of regressing the parameters i from the training data D,
where D is composed of a series of input x and their corresponding
labels y. The standard approach is to approximate a
minimal cost point estimate of the network parameters
i ,t
i.e., a
single value for each parameter (Figure 3a), using the backpropagation
algorithm, with all other possible parametrizations
of the network discarded. The cost function is often defined as
the log likelihood of the training set, sometimes with a regularization
term included. From a statistician's point of view, this is
a maximum likelihood estimation (MLE), or a maximum a
posteriori (MAP) estimation when regularization is used.
The point estimate approach, which is the traditional approach
in deep learning, is relatively easy to deploy with modern algorithms
and software packages, but tends to lack explainability [14].
The final model might also generalize in unforeseen and overconfident
ways on out-of-training-distribution data points [15], [16].
This property, in addition to the inability of ANNs to say " I don't
know " , is problematic for many critical applications. Of all the
techniques that exist to mitigate this [17], stochastic neural networks
have proven to be one of the most generic and flexible.
Stochastic neural networks are a type of ANN built by
(). Traditional ANNs such as
feedforward networks and recurrent networks are built using
one input layer
l ,, ,,
total number of layers.) In the simplest architecture of feedforward
networks, each layer l is represented as a linear transformation,
followed by a nonlinear operation s, also known as an
activation function:
lx
0 = ,
s
n
lWlb
yl
ii ii-1
=+
=
() [, ],
.
i 6 !
in1
(2)
introducing stochastic components into the network. This is
performed by giving the network either a stochastic activation
(Figure 3b) or stochastic weights (Figure 3c) to simulate multiple
possible models i with their associated probability distribution
().p i
ensemble learning [18].
The main motivation behind ensemble learning comes from
the observation that aggregating the predictions of a large set of
average-performing but independent predictors can lead to better
predictions than a single well-performing expert predictor
[19], [20]. Stochastic neural networks might improve their performance
over their point estimate counterparts in a similar fashion,
but this is not their main aim. Rather, the main goal of using
Stochastic Model
p(yx, q)
Prior
p(q)
Variational Posterior
(If Needed)
qφ(q)
Functional Model
y = Φθ(x)
Training Data
D = {(x1, y1), ..., (xn, yn)}
(a)
(b)
FIGURE 2 Workflow to design (a), train (b) and use a BNN for predictions (c).
(c)
Inference (Training)
MCMC
Gibbs Sampling,
Metropolis Hasting,
HMC, NUTS, ...
SGLD,
RECAST, ...
Variational Inference
SVI,
Bayes by Backprop,
Probabilistic
Backpropagation ...
MC-Dropout,
Deep Ensembles,
KFAC, SWAG ...
Posterior
p(qD)
Marginal
p(yx, D)
Input
x
Thus, BNNs can be considered a special case of
Summary
MAP
y
Uncertainty
Σyx, D
...
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 31
DL-Specific
Generic
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
