IEEE Computational Intelligence Magazine - May 2022 - 37

x
θ
x
x
D
y
y˜
(a)
θ
x
x′
y
D
(b)
y
(c)
D
y
T
(d)
yp
(e)
FIGURE 6 Different examples of PGMs to adapt the learning strategy for a given BNN (with stochastic weights). (a) Noisy labels, (b) Semi-supervised
learning, (c) Data augmentation, (d) Meta-learning, and (e) Self-supervised learning.
in designing or interpreting learning strategies. In particular,
the formulation of the Bayesian posterior, which is derived
from the different PGMs presented in Figure 6, can also be
used for a point estimate neural network to obtain a suitable
loss function to search for an MAP estimator for the parameters
(Section IV-C3). We also provide a practical example in the
Supplementary Material (Practical Example II) to illustrate
how such strategies can be implemented for an actual BNN.
1) Noisy Labels and Semi-Supervised Learning
The inputs Dx
because the labels Dy
in the training sets can be uncertain, either
are corrupted by noise [56], or because
labels are missing for a number of points. In the case of noisy
labels, one should extend the BBN to add a new variable for
the noisy labels yu conditioned on y (Figure 6a). It is common,
as the noise level itself is often unknown, to add a variable v to
characterize the noise. Frenay et al. [57] proposed a taxonomy
of the different approaches used to integrate v in a PGM (Figure
7). They distinguish three cases: noise completely at random
(NCAR); noise at random (NAR); and noise not at
random (NNAR) models. In the NCAR model, the noise v is
independent of any other variable, i.e., it is homoscedastic. In
the NAR model, v is dependent on the true label y but
remains independent of the features. NNAC models also
account for the influence of the features x, e.g., if the level of
noise in an image increases, then the probability that the image
has been mislabeled also increases. Both NAR and NNAC
models represent heteroscedastic, i.e., the antonym of homoscedastic,
noise.
These noise-aware PGMs are slightly more complex than a
purely supervised BNN, as presented in Section IV-B. However,
they can be treated in a similar fashion by deriving the formula
for the posterior from the PGM (Equation (12)) and
applying the chosen inference algorithm. For the NNAR
model, the most generic stochastic model of the three
described above (since the NCAR and NAR models are special
cases of the NNAR model), the posterior becomes:
pD pD pD pD pyx x ii
(, ,) (, )(yy ,) (, )( ). (28)
vi;? ;; ;
u vv
yy
D
x
σ y
y˜
(a)
During the prediction phase, y and v can simply be discarded
for each tuple (, ,)y vi sampled from the posterior.
In the case of partially labeled data (Figure 6b), also
known as semi-supervised learning, the dataset D is split into
labeled L and unlabeled U examples. In theory, this PGM can
be considered equivalent to the one used in the supervised
learning case depicted in Figure 5a, but in this case the unobserved
data U would bring no information. The additional
information of unlabeled data comes from the prior and only
the prior. Similar to traditional machine learning, the most
common approaches to implement semi-supervised learning in
Bayesian learning are either to use some type of data-driven
regularization [58] or to rely on pseudo labels [59].
Data-driven regularization implies modifying the prior
assumptions, and thus
the stochastic model, to be able to
extract meaningful information from the unlabeled dataset U.
There are two common ways to approach this process. The
first one is to condition the prior distribution of the model
parameters on the unlabeled examples to favor certain properties
of the model, such as a decision boundary in a low density
region, using a distribution ()
pU;i
yx
instead of ().p i This
implies formulating the stochastic model as:
() (, )( ),
pD pL Lp Uii i
;? ;;
where () is a prior with a consistency condition, as
defined in Equation (27). The consistency condition usually
pU;i
θ
D
x
σ y
y˜
(b)
θ
D
x
σ y
y˜
(c)
FIGURE 7 BBNs corresponding to (a) the noise completely at random
(NCAR), (b) noise at random (NAR) and (c) noise not at random
(NNAR) models from [57].
θ
(29)
yt
θ
θ
θp
l
θt
ξ
D
x
θs
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 37
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
