IEEE Computational Intelligence Magazine - May 2022 - 36

... regularization acts as a soft constraint on the
search space, in a manner similar to what a prior
does for a posterior.
one hidden layer. This is achieved by permuting two rows in
(, ),Wbii
the weights and their corresponding bias ,bi
i 1+
of one of
the hidden layers as well as the corresponding columns in the
following layer's weight matrix
W . This means that as the
number of hidden layers and the number of units in the
hidden layers grow, the number of equivalent representations,
which would roughly correspond to the modes
in
the posterior distribution, grows factorially. A mitigation
strategy is to enforce the bias vector in each layer to be
sorted in an ascending or a descending order. However, the
practical effects of doing so may be to degrade optimization:
weight-space symmetry may implicitly support the
exploration of the parameter space during the early stages
of the optimization.
Scaling symmetry is an unidentifiability problem arising
when using nonlinearities with the property () (),
sx sxaa=
which is the case of RELU and Leaky-RELU, two popular
nonlinearities in modern machine learning. In this case, assigning
the weights
WW ,ll 1+ to two consecutive layers l and l + 1
aa 1+ll1 ,( /)
becomes strictly equivalent to assigning WW . This
can reduce the convergence speed for point estimate neural
networks, a problem that is addressed in practice with various
activation normalization techniques [54]. BNNs are slightly
more complex as the scaling symmetry influences the posterior
shape, making it harder to approximate. Givens transformations
(also called Givens rotations) have been proposed as a mean to
constrain the norm of the hidden layers [53] and address the
scaling symmetry issue. In practice, using a Gaussian prior
already reduces the scaling symmetry problem, as it favors
weights with the same Frobenius norm on each layer. A soft
version of the activation normalization can also be implemented
by using a consistency condition; see Section IV-C4. The
additional complexity associated with sampling the network
parameters in a constrained space to perfectly remove the scaling
symmetry is computationally prohibitive. We provide, in the
Practical Example III of the Supplementary Material, additional
discussion on this issue using the " Paperfold " practical example.
3) The Link Between Regularization and Priors
The usual learning procedure for a point estimate neural network
is to find the set of parameters i that minimize a loss
function built using the training set D:
t
ii=
i
argmin lossDD,xy ( ).
(21)
Assuming that the loss is defined as minus the log-likelihood
function up to an additive constant, the problem can be rewritten
as:
t = argmax pD Dyx,),
(
ii;
i
(22)
36 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
t
which would be the first half of the model
according to the Bayesian paradigm. Now, assume
that we also have a prior for
i , and we want to
find the most likely point estimate from the posterior.
The problem can be reformulated as:
t = argmax pD Dp i
(
ii;yx,) ().
i
Next, one would go back to a log-likelihood formulation:
( )( ),
ii i=+
i
argmin lossDD,xy
reg
(23)
(24)
which is easier to optimize. Equation (24) is how regularization
is usually applied in machine learning and in many other
fields. Another argument, less formal, is that regularization acts
as a soft constraint on the search space, in a manner similar to
what a prior does for a posterior.
4) Prior With a Consistency Condition
Regularization can also be implemented with a consistency
condition (, ),xC i
which is a function used to measure how
well the model respects some hypothesis given a parametrization
i and an input x. For example, C can be set to favor
sparse or regular predictions to encourage monotonicity of
predictions with respect to some input variables (e.g., the probability
of getting the flu increases with age), or to favor decision
boundaries in low density regions when using semi-supervised
learning; see Section IV-D1. C can be seen as the relative log
likelihood of a prediction given the input x and parameter set
i . Thus, it can be included in the prior. To this end, C should
be averaged over all possible inputs:
CC(, )( ).xx xpdii= #
()
x
In practice, as ()xp is unknown, ()C i is approximated from the
features in the training set:
C () .
ii
!
;;
Dx
C
1 / (, ).x
x Dx
(26)
We can now write a function proportional to the prior with
the consistency condition included:
^ ;? exp - 1 / C^i ,,
pD p^ hii c ;; x Dx
xh
Dx
!
xhm
where ()p i is the prior without the consistency condition.
D. Degree of Supervision and Alternative
Forms of Prior Knowledge
The architecture presented in Section IV-B focuses mainly on
the use of BNNs in a supervised learning setting. However, in
real world applications, obtaining ground-truth labels can be
expensive. Thus, new learning strategies should be adopted
[55]. We will now present how to adapt BNNs for different
degrees of supervision. While doing so, we will also demonstrate
how PGMs in general and BBNs in particular are useful
(27)
(25)
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
