IEEE Computational Intelligence Magazine - August 2020 - 18

III. Types of Dataset Shifts

nn

A. Covariate Shift

A covariate shift is observed when future values of covariates x
differ from past observations [16], [17]. Covariate shifts occur
only when the distribution of x suffers a drift and all other
probabilities remain unchanged, thus, Ptrain (y|x) = Ptest (y|x) but
Ptrain (x) ! Ptest (x). Therefore, the relation between the target y
and the covariate x remains the same, but the covariate distribution P (x) changes, as shown in the histograms in Fig. 1(a). A
purely discriminative solution for classification problems with
covariate shift can be implemented by searching for all model
parameters according to an integrated optimization approach
[18]. Such a solution shows that exponential models, such as
Gaussian kernels, lead to a convex setting, which allows for
simplified optimization with Newton gradient descent methods, resulting in a kernel logistic regression and an exponential
model classifier.
In eq. 2, the risk changes with covariate shift as the integration of the loss function L is performed with respect to a different input space. If the model risk is calculated over m
different datasets, then it can be understood as a random variable R. Considering the Central Limit Theorem and that R has
a quadratic loss function, the risk mean and variance are given
by eq. 5 and 6.
E 6R@ = E 6E 6(y - f (x, w)) 2@@ = m - 1 n 2
(5)
m
(m -1) 2
(m -1)(m - 3) 2
var (R) = var 6E 6(y - f (x, w)) 2@@ =
n4 n2
m3
m3
(6)
where

10

=

# (y - f (x, w)) n dP (x, y)

(7)

and P (x, y) = P (y|x) P (x) is the joint probability density. Thus,
the expected value for risk might change and its variance can
increase, resulting in a larger error due to dependencies on P (x).
A typical example associated with covariate shift is the braincomputer interface (BCI), which is based on electroencephalograms (EEG). In EEGs, electrode placement, attention level, user
fatigue, and other factors influence brain activity scanning, which
causes signals to be highly variable [20]-[22] Due to the complexity of training protocols and pre-processing procedures, BCI
systems are often trained off-line. Moreover, adaptive classifiers
and transfer learning tend to perform well [23]. The example in
Fig. 1 is from the Dataset IVc of the BCI Competition III [19],
[24]. Training and test sets were sampled at different times, consisting of discriminating between the motor imagery of left hand
movement and right foot movement using EEG signals. Ensemble-based classifiers used to solve this problem tend to outperform single models when they are segmented into local models
according to the covariate input space [23], [25].
B. Prior Probability Shift

Prior probability shifts occur when the target distribution P (y)
differs between training and test data, affecting models that
have assumptions of causal relationships in data. This is relevant
in models that infer conditional probability P (y|x) through
P (x|y) P (y), such as naive Bayes classifiers, which consider the
conditional probability of the training set to classify the test
data. If covariates x depend on targets y and the target distribution P (y) shifts, the results given by the model might
differ from expected values [9], [10], [26], [27]. In classification

CSP0

CSP1

CSP2

Classes
Left x Foot
240
160
80
0
-80
-160
-240

0
6
4
(b)

2
0
CSP0
-2

CSP1

CSP2

Left_Train
Foot_Train
Left_Test
Foot_Test

-4

-5

0
(a)

5

0

Classes
Left x Foot
400
200
0
-200
-400

10
(c)

FIGURE 1 The spatial distribution of brain activity represented with common spatial patterns (CSP) of each class differs greatly between training
and test data [19]. (a) Spatial distribution of the two principal components for both classes in training and test sets. (b) Training set spatial representation of brain activity. (c) Test set spatial representation of brain activity.

18

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2020
IEEE Computational Intelligence Magazine - August 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2020
