IEEE Computational Intelligence Magazine - August 2020 - 18
III. Types of Dataset Shifts
nn
A. Covariate Shift
A covariate shift is observed when future values of covariates x
differ from past observations [16], [17]. Covariate shifts occur
only when the distribution of x suffers a drift and all other
probabilities remain unchanged, thus, Ptrain (y|x) = Ptest (y|x) but
Ptrain (x) ! Ptest (x). Therefore, the relation between the target y
and the covariate x remains the same, but the covariate distribution P (x) changes, as shown in the histograms in Fig. 1(a). A
purely discriminative solution for classification problems with
covariate shift can be implemented by searching for all model
parameters according to an integrated optimization approach
[18]. Such a solution shows that exponential models, such as
Gaussian kernels, lead to a convex setting, which allows for
simplified optimization with Newton gradient descent methods, resulting in a kernel logistic regression and an exponential
model classifier.
In eq. 2, the risk changes with covariate shift as the integration of the loss function L is performed with respect to a different input space. If the model risk is calculated over m
different datasets, then it can be understood as a random variable R. Considering the Central Limit Theorem and that R has
a quadratic loss function, the risk mean and variance are given
by eq. 5 and 6.
E 6R@ = E 6E 6(y - f (x, w)) 2@@ = m - 1 n 2
(5)
m
(m -1) 2
(m -1)(m - 3) 2
var (R) = var 6E 6(y - f (x, w)) 2@@ =
n4 n2
m3
m3
(6)
where
10
=
# (y - f (x, w)) n dP (x, y)
(7)
and P (x, y) = P (y|x) P (x) is the joint probability density. Thus,
the expected value for risk might change and its variance can
increase, resulting in a larger error due to dependencies on P (x).
A typical example associated with covariate shift is the braincomputer interface (BCI), which is based on electroencephalograms (EEG). In EEGs, electrode placement, attention level, user
fatigue, and other factors influence brain activity scanning, which
causes signals to be highly variable [20]-[22] Due to the complexity of training protocols and pre-processing procedures, BCI
systems are often trained off-line. Moreover, adaptive classifiers
and transfer learning tend to perform well [23]. The example in
Fig. 1 is from the Dataset IVc of the BCI Competition III [19],
[24]. Training and test sets were sampled at different times, consisting of discriminating between the motor imagery of left hand
movement and right foot movement using EEG signals. Ensemble-based classifiers used to solve this problem tend to outperform single models when they are segmented into local models
according to the covariate input space [23], [25].
B. Prior Probability Shift
Prior probability shifts occur when the target distribution P (y)
differs between training and test data, affecting models that
have assumptions of causal relationships in data. This is relevant
in models that infer conditional probability P (y|x) through
P (x|y) P (y), such as naive Bayes classifiers, which consider the
conditional probability of the training set to classify the test
data. If covariates x depend on targets y and the target distribution P (y) shifts, the results given by the model might
differ from expected values [9], [10], [26], [27]. In classification
CSP0
CSP1
CSP2
Classes
Left x Foot
240
160
80
0
-80
-160
-240
0
6
4
(b)
2
0
CSP0
-2
CSP1
CSP2
Left_Train
Foot_Train
Left_Test
Foot_Test
-4
-5
0
(a)
5
0
Classes
Left x Foot
400
200
0
-200
-400
10
(c)
FIGURE 1 The spatial distribution of brain activity represented with common spatial patterns (CSP) of each class differs greatly between training
and test data [19]. (a) Spatial distribution of the two principal components for both classes in training and test sets. (b) Training set spatial representation of brain activity. (c) Test set spatial representation of brain activity.
18
IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | AUGUST 2020
IEEE Computational Intelligence Magazine - August 2020
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2020
Contents
IEEE Computational Intelligence Magazine - August 2020 - Cover1
IEEE Computational Intelligence Magazine - August 2020 - Cover2
IEEE Computational Intelligence Magazine - August 2020 - Contents
IEEE Computational Intelligence Magazine - August 2020 - 2
IEEE Computational Intelligence Magazine - August 2020 - 3
IEEE Computational Intelligence Magazine - August 2020 - 4
IEEE Computational Intelligence Magazine - August 2020 - 5
IEEE Computational Intelligence Magazine - August 2020 - 6
IEEE Computational Intelligence Magazine - August 2020 - 7
IEEE Computational Intelligence Magazine - August 2020 - 8
IEEE Computational Intelligence Magazine - August 2020 - 9
IEEE Computational Intelligence Magazine - August 2020 - 10
IEEE Computational Intelligence Magazine - August 2020 - 11
IEEE Computational Intelligence Magazine - August 2020 - 12
IEEE Computational Intelligence Magazine - August 2020 - 13
IEEE Computational Intelligence Magazine - August 2020 - 14
IEEE Computational Intelligence Magazine - August 2020 - 15
IEEE Computational Intelligence Magazine - August 2020 - 16
IEEE Computational Intelligence Magazine - August 2020 - 17
IEEE Computational Intelligence Magazine - August 2020 - 18
IEEE Computational Intelligence Magazine - August 2020 - 19
IEEE Computational Intelligence Magazine - August 2020 - 20
IEEE Computational Intelligence Magazine - August 2020 - 21
IEEE Computational Intelligence Magazine - August 2020 - 22
IEEE Computational Intelligence Magazine - August 2020 - 23
IEEE Computational Intelligence Magazine - August 2020 - 24
IEEE Computational Intelligence Magazine - August 2020 - 25
IEEE Computational Intelligence Magazine - August 2020 - 26
IEEE Computational Intelligence Magazine - August 2020 - 27
IEEE Computational Intelligence Magazine - August 2020 - 28
IEEE Computational Intelligence Magazine - August 2020 - 29
IEEE Computational Intelligence Magazine - August 2020 - 30
IEEE Computational Intelligence Magazine - August 2020 - 31
IEEE Computational Intelligence Magazine - August 2020 - 32
IEEE Computational Intelligence Magazine - August 2020 - 33
IEEE Computational Intelligence Magazine - August 2020 - 34
IEEE Computational Intelligence Magazine - August 2020 - 35
IEEE Computational Intelligence Magazine - August 2020 - 36
IEEE Computational Intelligence Magazine - August 2020 - 37
IEEE Computational Intelligence Magazine - August 2020 - 38
IEEE Computational Intelligence Magazine - August 2020 - 39
IEEE Computational Intelligence Magazine - August 2020 - 40
IEEE Computational Intelligence Magazine - August 2020 - 41
IEEE Computational Intelligence Magazine - August 2020 - 42
IEEE Computational Intelligence Magazine - August 2020 - 43
IEEE Computational Intelligence Magazine - August 2020 - 44
IEEE Computational Intelligence Magazine - August 2020 - 45
IEEE Computational Intelligence Magazine - August 2020 - 46
IEEE Computational Intelligence Magazine - August 2020 - 47
IEEE Computational Intelligence Magazine - August 2020 - 48
IEEE Computational Intelligence Magazine - August 2020 - 49
IEEE Computational Intelligence Magazine - August 2020 - 50
IEEE Computational Intelligence Magazine - August 2020 - 51
IEEE Computational Intelligence Magazine - August 2020 - 52
IEEE Computational Intelligence Magazine - August 2020 - 53
IEEE Computational Intelligence Magazine - August 2020 - 54
IEEE Computational Intelligence Magazine - August 2020 - 55
IEEE Computational Intelligence Magazine - August 2020 - 56
IEEE Computational Intelligence Magazine - August 2020 - 57
IEEE Computational Intelligence Magazine - August 2020 - 58
IEEE Computational Intelligence Magazine - August 2020 - 59
IEEE Computational Intelligence Magazine - August 2020 - 60
IEEE Computational Intelligence Magazine - August 2020 - 61
IEEE Computational Intelligence Magazine - August 2020 - 62
IEEE Computational Intelligence Magazine - August 2020 - 63
IEEE Computational Intelligence Magazine - August 2020 - 64
IEEE Computational Intelligence Magazine - August 2020 - 65
IEEE Computational Intelligence Magazine - August 2020 - 66
IEEE Computational Intelligence Magazine - August 2020 - 67
IEEE Computational Intelligence Magazine - August 2020 - 68
IEEE Computational Intelligence Magazine - August 2020 - 69
IEEE Computational Intelligence Magazine - August 2020 - 70
IEEE Computational Intelligence Magazine - August 2020 - 71
IEEE Computational Intelligence Magazine - August 2020 - 72
IEEE Computational Intelligence Magazine - August 2020 - 73
IEEE Computational Intelligence Magazine - August 2020 - 74
IEEE Computational Intelligence Magazine - August 2020 - 75
IEEE Computational Intelligence Magazine - August 2020 - 76
IEEE Computational Intelligence Magazine - August 2020 - Cover3
IEEE Computational Intelligence Magazine - August 2020 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com