IEEE Computational Intelligence Magazine - May 2022 - 38

expresses the fact that points that are close to each other should
lead to the same prediction, e.g., graph Laplacian norm regularization
[60].
The second way is to assume some kind of dependency
across the observed and unobserved labels in the dataset. This
type of semi-supervised Bayesian learning relies either on an
undirected PGM [61] to build the prior or at least does not
assume independence between different training pairs (, )xy
[62]. To keep things simple, we represent this fact by dropping
the plate around y in Figure 6b. The posterior is written in the
usual way (Equation (4)). The main difference is that
(, )
pD Dyx i is chosen to enforce some kind of consistency
;
across the dataset. For example, one can assume that two close
points are likely to have similar labels y with a level of uncertainty
that increases with the distance.
Both approaches have a similar effect and the choice of one
over the other will depend on the mathematical formulation
favored to build the model.
The semi-supervised learning strategy can also be reformulated
as having a weak predictor capable of generating some
pseudo labels y,u
Many of the algorithms used for semi-supervised learning use
an initial version of the model trained with the labeled examples
[63] to generate the pseudo labels yu
and train the final
model with y .u This is problematic for BNNs. When the prediction
uncertainty is accounted for, reducing the uncertainty
associated with the unlabeled data becomes impossible, at least
not without an additional hypothesis in the prior. Even if it is
less current in practice, using a simpler model [64] to obtain
the pseudo labels can help mitigate that problem.
2) Data Augmentation
Data augmentation in machine learning is a strategy that is used
to significantly increase the diversity of the data D available to
train deep models, without actually collecting new data. It relies
on transformations that act on the input but have no or very low
probability to change the label (or at least do so in a predictable
way) to generate an augmented dataset A(D). Examples of such
transformations include applying rotations, flipping or adding
noise in the case of images. Data augmentation is now at the
forefront of state-of-the-art techniques in computer vision [59]
and increasingly in natural language processing [65].
The augmented dataset A(D) could contain an infinite set
of possible variants of the initial dataset D, e.g., when using
continuous transforms such as rotations or additional noise. To
achieve this in practice, A(D) is sampled on the fly during
training, rather than generating in advance all possible augmentations
in the training set. This process is straightforward when
training point estimate neural networks, but there are some
subtleties when applying it in Bayesian statistics. The main concern
is that the posterior of interest is (, ),pD Aug
z;
pA DD;z
where
Aug represents some knowledge about augmentation, not
(( ), ),
since A(D) is not observed. From a Bayesian
perspective, the additional information is brought by the
knowledge of the augmentation process rather than by some
38 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
additional data. Stated otherwise, the data augmentation is a
part of the stochastic model (Figure 6c).
The idea is that if one is given data D, then one could also
have been given data D ,l where each element in D is replaced
by an augmentation. Then, Dl is a different perspective of the
data D. To model this, we have the augmentation distribution
(, )xx
pAug that augments the observed data using the augmentation
model Aug to generate (probabilistically) x ,l which
l ;
represents data in the vicinity of x (Figure 6c). xl can then be
marginalized to simplify the stochastic model. The posterior is
given by:
pAug pp Augdx p`j#;? ;;
xy
xl
yx
(, ,) (, )(xx,) ().
ii i
ll l
(30)
This is a probabilistic counterpart to vicinal risk [66].
The integral in Equation (30) can be approximated using
Monte Carlo integration by sampling a small set of augmentations
Ax
according to (, ) and averaging:
py xAug . 1 / py x;;l(, ).
px xAug
l ;
(, ,)
sometimes with some confidence level.
ii
!
;; l
A
x xAx
When training using a Monte-Carlo-based estimate of the loss,
Ax
(31)
can contain as few as a single element as long as it is resampled
for each optimization iteration. This greatly simplifies the
evaluation of Equation (31).
An extension of this approach works in the context of
semi-supervised learning. The prior can be designed to encourage
consistency of predictions under augmentation [59], [67],
using unlabeled data to build the samples for the consistency
condition, as defined in Equation (27). Note that this does not
add labeling to the unlabeled examples but only adds a term to
encourage consistency between the labels for an unlabeled data
point and its augmentation.
3) Meta-Learning, Transfer Learning, and
Self-Supervised Learning
Meta-learning [68], in the broadest sense, is the use of
machine learning algorithms to assist in the training and optimization
of other machine learning models. The meta knowledge
acquired by meta-learning can be distinguished from
standard knowledge in the sense that it is applicable to a set of
related tasks rather than a single task.
Transfer learning designates methods that reuse some
intermediate knowledge acquired on a given problem to address
a different problem. In deep learning, it is used mostly for
domain adaptation, when labeled data are abundant in a domain
that is in some way similar to the domain of interest but scarce in
the domain of interest [69]. Alternatively, pre-trained models [70]
could be used to study large architectures whose complete training
would be very computationally expensive.
Self-supervised learning is a learning strategy where the
data themselves provide the labels [71]. Since the labels directly
obtainable from the data do not match the task of interest, the
problem is approached by learning a pretext (or proxy) task in
addition to the task of interest. The use of self-supervision is now
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
