IEEE Computational Intelligence Magazine - August 2020 - 20

❏ Virtual concept drift: The joint probability P (y, x) differs

between training and test data because Ptrain (x) ! Ptest (x),
which is a covariate shift.
Smart industries have contributed to an increase in the
number of sensors in machines, which allows self-sensing and
integration with data analytics tools [37]. Smart monitoring
and control of machinery tend to become increasingly important with the development of Industry 4.0. In this context, the
example in Fig. 3 shows how the speed of motors varies
according to a different operation point or phase fault. In this
case, dataset shift is considered when the motor control system
should react to different working conditions and possible faults.

A. Sample Selection Bias

A common cause of dataset shift is the selection of uniform, or
biased, training sets. Consider that the training data is chosen
according to a sampling decision variable s and its probability

45

Albumin

40
35
Train Die
Train Live
Test Die

25

Test Live
0

25
50
75
Prothrombin Time
(a)

100

45

Albumin

40
35
Train Die
30

Train Live
Test Die

25

Test Live
0

25
50
75
Prothrombin Time
(b)

100

FIGURE 4 Projection of the features albumin and prothrombin time
from the hepatitis dataset. Class Live (green) has significantly more
instances than class Die (red) in (a). The undersampling process that
was dependent on values of other features, such as malaise, age, and
histology, resulted in the misspecified models in (b), with the class
contours significantly different. (a) Original imbalanced datasets with
a Gaussian SVM classifier, with 85% test accuracy (black border dots).
(b) Undersampled, rebalanced data with a Gaussian SVM classifier,
with 63% test accuracy.

20

Ptrain (y, x) = P (y, x|s = 1)
= P (s = 1|y, x) P (y|x) P (x).
Ptest (y, x) = P (y, x)
= P (y|x) P (x).
Ptrain (y, x) ! Ptest (y, x).

IV. Causes of Dataset Shift

30

density is given by eq. 8. In this equation, the decision variable
is dependent on targets y and covariates x, which causes a bias
in the data used for training and, therefore, also on the probability density Ptrain. Meanwhile, test data and its probability
density Ptest are not subject to this same bias, which reflects a
shift between training and test data, as represented in eq. 8 to
10 [9], [10].

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2020

(8)
(9)
(10)

In this case, the sample is selected when s = 1 and discarded
when s = 0. The types of shift that might occur in this scenario
are: (1) covariate shift when Ptrain (s = 1|x); (2) prior pr shift
when Ptrain (s = 1|y); and (3) any type when Ptrain (s = 1|y, x),
which characterizes missing not at random (MNAR) sampling
systems [9], [10].
Bias in estimators may be induced by unequal selection probabilities at any stage of sampling. Consistent estimators can be
obtained by weighting the model estimation with reciprocals of
selection probabilities at each stage [38]. Bias problems are often
seen in off-line scenarios, where labeled data is difficult to obtain,
such as diagnostic and clinical studies [39]-[42]. In these cases,
training data can be biased because exams are often performed
only on diseased subjects and healthy subjects are undersampled.
Furthermore, prior probability may differ between training and
test data if the criteria used to choose patients changes.
Differences in balance between training and test sets in classification problems are a prior probability shift scenario. Thus,
methods to solve this shift can be used in imbalanced datasets
problems alongside with rebalancing strategies [26], [27]. For
example, class distribution estimation can be solved by quantification learning methods [7]. However, appropriate balancing
strategies should be used to guarantee the model quality. For
instance, in Fig. 4, the hepatitis dataset [43] was rebalanced with
an inadequate MNAR approach and resulted in a poor model.
To minimize possible dataset shifts, classes can be balanced
through a partition of the dataset using "Distribution optimally
balanced stratified cross-validation" [41]. Likewise, intrinsic characteristics of the data can be used to rebalance data [44].
B. Domain Shift

An example of domain shift is illustrated in Fig. 5(a), where the
covariate x is not the actual latent variable x0, but instead is an
observation given by a function x = f (x 0). When the function
f (x 0) is altered, the covariate x perceived by the model is different even if the latent variable x0 remains the same. Thus, domain
shift characterizes changes in the measurement system, metrics,
or even the description of the data generator, such as currency
devaluation in pricing predictions or the visual classification of
images with different lighting. In image classification, inputs
are pictures from real-world scenes and capture methods are
IEEE Computational Intelligence Magazine - August 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2020
