IEEE Computational Intelligence Magazine - May 2022 - 34

learning, previous posteriors can be recycled as priors when
new data become available to avoid the so-called problem of
catastrophic forgetting [44]; see Algorithm 3.
IV. Setting the Stochastic Model for a
Bayesian Neural Network
Designing a BNN requires choosing a functional model and
a stochastic model. This tutorial will not cover the design of
the functional model, as almost any model used for point
estimate networks can be used as a functional model
for a BNN. Furthermore, a rich literature on the subject exists
already; see, for example, [45]. Instead, this section will focus on
how to design the stochastic model. Section IV-A introduces
probabilistic graphical models (PGMs), a tool used to represent
the relationships between the model's stochastic variables. Section
IV-B details how to derive the posterior for a BNN from
its PGM. Section IV-C discusses how to choose the probability
laws used as priors. Finally, Section IV-D presents how the
choice of a PGM can affect the degree of supervision or incorporate
other forms of prior knowledge into the model.
A. Probabilistic Graphical Models
Probabilistic graphical models (PGMs) use graphs to represent
the interdependence of multivariate stochastic variables and
subsequently decompose their probability distributions. PGMs
cover a large variety of models. The type of PGMs this tutorial
focuses on are Bayesian belief networks (BBN), which are
PGMs whose graphs are acyclic and directed. We refer the
reader to [46] for more details on how to represent learning
algorithms using general PGMs.
In a PGM, variables vi
are the nodes in the graph. Different
symbols are used to distinguish the nature of the considered variables
(Figure 4). A directed link, which is the only type of link
allowed in a BBN, means that the probability distribution of the
Algorithm 3 Online learning loop with a BNN.
Define
pp ;0ii=^^
hh
while true do
Define
^i;
end while
Definepp ;Di;
pDh =
^i =+hi 1
i
^
ih
y
i $
pD Dp d
pD Dp
^ ii;
^ ii;
yx
yx
,,
,,
, ll l
ii
i
,
h
h
^
^
hi i
ihi
;
pp
() (, ).
vv vv
latent obsobs latent
;?
The joint distribution (, )vvp obslatent
ent inference algorithms; see Section V.
(12)
is then used by the differB.
Defining the Stochastic Model of a BNN From a PGM
Consider the two models presented in Figure 5, with both the
BNN and the corresponding BBN depicted. The BNN with
stochastic weights (Figure 5a), if meant to perform regression,
could represent the following data generation process:
+;
ii
yy Nxx
p
+ () =
p
y
(a)
θ
(b)
l
(c)
v
a
B
(d)
(e)
FIGURE 4 The different symbols PGM, (a) observed variables are in
colored circles, (b) unobserved variables are in white circles, (c)
deterministic variables are in dashed circles and (d) parameters are
in rectangles. Plates, represented as a rectangle around a subgraph,
indicate multiple independent instances of the subgraph for a batch
of variables B (e).
n R
i =
Ui
(, )( (),).
N(, ),
R
(13)
The choice of using normal laws N( ,),n R with mean n and
covariance ),R is arbitrary but is common in practice because
of its good mathematical properties.
For classification, the model samples the prediction from a
categorical law (),pCat
i
+;
ii
i.e.,
yyxx
p
+ () =
p
n R
i = Cat Ui
(, )( ()).
N(, ),
(14)
Then, one can use the fact that multiple data points from the
training set are independent, as indicated by the plate notation
in Figure 5, to write the probability of the training set as:
34 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
target variable is defined conditioned on the source variable. The
fact that the BBN is acyclic allows the computation of the joint
probability distribution of all the variables vi
n
in the graph:
pp parents
i =1
1
n
;
(,f,) (( )).
vv vv= % ii;
(10)
The type of distribution used to define the conditional probabilities
(( ))vvp parentsii
depends on the context. Once the
conditional probabilities are defined, the BBN describes a data
generation process. Parents are sampled before their children.
This is always possible since the graph is acyclic. All the variables
together represent a sample from the joint probability distribution
(, ,).vvp
1
f
n
Models usually learn from multiple examples sampled from
the same distribution. To highlight this fact, the plate notation
(Figure 4e) has been introduced. A plate indicates that the
variables (, ,)
vvn in the subgraph encapsulated by the plate
1
f
are copied along a given batch dimension. A plate implies independence
between all the duplicated nodes. This fact can be
exploited to compute the joint probability of a batch
(, ,) :, ,
Bb Bff;;
== as:
()
{}
pB =
vvnb
1
1
(, ,)f !vv B
1
% p(,f,).vv
n
1
n
(11)
In a PGM, the observed variables, depicted in Figure 4a using
colored circles, are treated as the data. The unobserved, also called
latent variables, represented by a white circle in Figure 4b, are
treated as the hypothesis. From the joint probability derived from
the PGM, defining the posterior for the latent variables given the
observed variables is straightforward using Bayes' formula:
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
