IEEE Computational Intelligence Magazine - May 2022 - 35

pD Dp
(, )
(, )( ,).yx
xy D
yx
;;
ii=
!
%
lx
ll l
yl
0 = ,
ii ii iips N(Wl b in1
+;
= n
() (
.
-1 =
-1 + i,))[ ,],
R
6 !
(16)
The formulation of the joint probability is slightly more complex
as we have to account for the chain of dependencies
spanned by the BBN over the multiple latent variables
l[,
pDlllDp[,n(,
) D i =1
^ yxh
11]
,. (17)
ll
;;=
!
0 n
c
% % ^ ii-1hm
n
It is sometimes possible, and often desirable, to define ()p ll
ii 1
;
W
b
lWlbs
bb
+
+ n
=+1
N(,
),
N
nWW
R
R
(, ),
()
is equivalent to sampling l as:
( (,() () )),
l N lI lI l++R
+ 77nn R-- -
<
s Wb Wb11 1
(18)
-
such that the BNNs described in Figure 5a and in Figure 5b
can be considered equivalent. For instance, sampling l as:
11n- ] :
2) Addressing Unidentifiability in
Bayesian Neural Networks
One of the main problems with Bayesian deep learning is that
deep neural networks are overparametrized models, i.e., they
have many equivalent parametrizations [52]. This is an example
of statistical unidentifiability, which can lead to complex
multimodal posteriors that are hard to sample and approximate
when training a BNN [22]. There are two solutions to
deal with this issue: (1) changing the functional model
parametrization, or (2) constraining the support of the prior
to remove unidentifiability.
The two most common classes of nonuniqueness in ANNs
(19)
where 7 denotes a Kronecker product.
The basic Bayesian regression architecture shown in Figure 5a
is more common in practice. The alternative architecture shown
in Figure 5b is sometimes used as it allows compressing the number
of variational parameters when using variational inference
[47]; see also Section V.
C. Setting the Priors
Setting the prior of a deep neural network is often not an intuitive
task. The main problem is that it is not truly explicit how
models with a very large number of parameters and a nontrivial
architecture such as an ANN will generalize for a given
parametrization [48]. In this Section, we first present the common
practice, discuss the issues related to the statistical
unidentifiability of ANNs, and then show the link between
the prior for BNNs and regularization for the point estimate
algorithms. Finally, we present a method to build the prior
from high level knowledge.
1) A good Default Prior
For basic architectures such as Bayesian regression (Figure 5a),
a standard procedure is to use a normal prior with a zero
mean 0 and a diagonal covariance Iv on the coefficients of
the network:
p N(, ).Iv0
()
i =
(20)
D
(a)
x
x
θ
are weight-space symmetry and scaling symmetry [53]. Both
are not a concern for point estimate neural networks but might
be for BNNs. Weight-space symmetry implies that one can
build an equivalent parametrization of an ANN with at least
(15)
In the case of stochastic activations (Figure 5b), the data generation
process might become:
This approach is equivalent to a weighted 2, regularization
(with weights /)1 v when training a point estimate network, as
will be demonstrated in Section IV-C3. The documentation of
the probabilistic programming language Stan [49] provides
examples on how to choose v knowing the expected scale of
the considered parameters [50].
Although such an approach is often used in practice, there is
no theoretical argument that makes it better than any other
formulation [51]. The normal law is preferred due to its mathematical
properties and the simple formulation of its log, which
is used in most of the learning algorithms.
y
lh
θ
D
(b)
FIGURE 5 BBNs with (a) coefficients as stochastic variables and
(b) activations as stochastic variables.
y
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 35
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
