Computational Intelligence - February 2016 - 42
Because classification and regression have
similarities and differences, there are common and specific ensemble methods for
classification and regression. It is therefore
more suitable to discuss the ensemble methods for classification and regression together.
The paper is organized as follows: in
Section II, we review the theories of
ensemble methods; Ensemble methods
for classification and regression are presented in Section III; In Section IV, we
summarize the paper. We suggest future
research directions in Section V.
Table 1 Nomenclature.
abbReVIaTION
DeFINITION
AdaBoost
ADAPTIVE BOOSTING
ANFIS
ADAPTIVE NEURO-FUZZY INFERENCE SYSTEM
ANN
ARTIFICIAL NEURAL NETWORK
ARCING
ADAPTIVELY RESAMPLE AND COMBINE
ARIMA
AUTO-REGRESSIVE INTEGRATED MOVING AVERAGE
ARTMAP
PREDICTIVE ADAPTIVE RESONANCE THEORY
Bagging
BOOTSTRAP AGGREGATION
CNN
CONVOLUTIONAL NEURAL NETWORK
DENFIS
DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM
DIVACE
DIVERSE AND ACCURATE ENSEMBLE LEARNING ALGORITHM
II. Theory
DNN
DEEP NEURAL NETWORK
The main theory behind ensemble methods is bias-variance-covariance decomposition. It offers theoretical justification for
improved performance of an ensemble
over its constituent base predictors. The
key to ensemble methods is diversity,
which includes data diversity, parameter
diversity, structural diversity, multi-objective optimization and fuzzy methods.
EMD
EMPIRICAL MODE DECOMPOSITION
GLM
GENERALIZED LINEAR MODELS
IMF
INTRINSIC MODE FUNCTION
KNN
K NEAREST NEIGHBOR
LR
LINEAR REGRESSION
LS-SVR
LEAST SQUARE SUPPORT VECTOR REGRESSION
MKL
MULTIPLE KERNEL LEARNING
MLMKL
MULTI-LAYER MULTIPLE KERNEL LEARNING
MLP
MULTIPLE LAYER PERCEPTRON
A. Bias-Variance-Covariance
Decomposition
MPANN
MEMETIC PARETO ARTIFICIAL NEURAL NETWORK
MPSVM
MULTI-SURFACE PROXIMAL SUPPORT VECTOR MACHINE
Researchers initially investigated the theory behind ensemble methods by using
regression problems. In the context of
regression, there is a theoretical proof to
show that a proper ensemble predictor can
guarantee to have smaller squared error
than the average squared error of the base
predictors.The proof is based on ambiguity decomposition [11], [12]. However,
ambiguity decomposition only applies to a
single dataset with ensemble methods. For
multiple datasets, bias-variance-covariance
decomposition is introduced [12]-[15]
and the equation is shown:
NCL
NEGATIVE CORRELATION LEARNING
PCA
PRINCIPAL COMPONENT ANALYSIS
PSO
PARTICLE SWARM OPTIMIZATION
QCQP
QUADRATICALLY CONSTRAINED QUADRATIC PROGRAM
SMO
SEQUENTIAL MINIMAL OPTIMIZATION
SVM
SUPPORT VECTOR MACHINE
SVR
SUPPORT VECTOR REGRESSION
RBFNN
RADIAL BASIS FUNCTION NEURAL NETWORK
RNN
RECURRENT NEURAL NETWORK
RT
REGRESSION TREE
RVFL
RANDOM VECTOR FUNCTIONAL LINK
E [ f - t] 2 = bias 2 + 1 var
M
+ c 1 - 1 m covar
M
1
bias =
/ ^E [fi] - t h
M i
var = 1
M
/ E [fi - E [fi]] 2
i
1
//
M (M - 1) i j ! i
E [fi - E [fi]] (f j - E [f j])
covar =
(3)
where t is the target and fi is the output from each model and M is the size
42
of ensemble. The error is composed of
the average bias (which measures the
average difference between the prediction of the base learner and the desired
output), plus a term involving their
average variance (which measures the
average variability of the base learners),
and the third term involving their average pairwise covariance term (which
measures the average pairwise difference
of different base learners).
There exist several theoretical
insights about the soundness of using
ensemble methods such as strength-correlation [3], stochastic discrimination
IEEE ComputatIonal IntEllIgEnCE magazInE | FEbruary 2016
[16] and margin theory [17]. They have
been shown to be equivalent to biasvariance-covariance decomposition [18].
From the equation, we can see the
term covar can be negative, which may
decrease the expected loss of the
ensemble while leaving bias and var
unchanged. Beside the covar, the number
of models also plays an important role.
As it increases, the proportion of the
variance in the overall loss vanishes
whereas the impor tance of the
covar iance increases. Overall, this
decomposition shows that if we are able
to design lowcorrelated individual
Table of Contents for the Digital Edition of Computational Intelligence - February 2016
Computational Intelligence - February 2016 - Cover1
Computational Intelligence - February 2016 - Cover2
Computational Intelligence - February 2016 - 1
Computational Intelligence - February 2016 - 2
Computational Intelligence - February 2016 - 3
Computational Intelligence - February 2016 - 4
Computational Intelligence - February 2016 - 5
Computational Intelligence - February 2016 - 6
Computational Intelligence - February 2016 - 7
Computational Intelligence - February 2016 - 8
Computational Intelligence - February 2016 - 9
Computational Intelligence - February 2016 - 10
Computational Intelligence - February 2016 - 11
Computational Intelligence - February 2016 - 12
Computational Intelligence - February 2016 - 13
Computational Intelligence - February 2016 - 14
Computational Intelligence - February 2016 - 15
Computational Intelligence - February 2016 - 16
Computational Intelligence - February 2016 - 17
Computational Intelligence - February 2016 - 18
Computational Intelligence - February 2016 - 19
Computational Intelligence - February 2016 - 20
Computational Intelligence - February 2016 - 21
Computational Intelligence - February 2016 - 22
Computational Intelligence - February 2016 - 23
Computational Intelligence - February 2016 - 24
Computational Intelligence - February 2016 - 25
Computational Intelligence - February 2016 - 26
Computational Intelligence - February 2016 - 27
Computational Intelligence - February 2016 - 28
Computational Intelligence - February 2016 - 29
Computational Intelligence - February 2016 - 30
Computational Intelligence - February 2016 - 31
Computational Intelligence - February 2016 - 32
Computational Intelligence - February 2016 - 33
Computational Intelligence - February 2016 - 34
Computational Intelligence - February 2016 - 35
Computational Intelligence - February 2016 - 36
Computational Intelligence - February 2016 - 37
Computational Intelligence - February 2016 - 38
Computational Intelligence - February 2016 - 39
Computational Intelligence - February 2016 - 40
Computational Intelligence - February 2016 - 41
Computational Intelligence - February 2016 - 42
Computational Intelligence - February 2016 - 43
Computational Intelligence - February 2016 - 44
Computational Intelligence - February 2016 - 45
Computational Intelligence - February 2016 - 46
Computational Intelligence - February 2016 - 47
Computational Intelligence - February 2016 - 48
Computational Intelligence - February 2016 - 49
Computational Intelligence - February 2016 - 50
Computational Intelligence - February 2016 - 51
Computational Intelligence - February 2016 - 52
Computational Intelligence - February 2016 - 53
Computational Intelligence - February 2016 - 54
Computational Intelligence - February 2016 - 55
Computational Intelligence - February 2016 - 56
Computational Intelligence - February 2016 - 57
Computational Intelligence - February 2016 - 58
Computational Intelligence - February 2016 - 59
Computational Intelligence - February 2016 - 60
Computational Intelligence - February 2016 - 61
Computational Intelligence - February 2016 - 62
Computational Intelligence - February 2016 - 63
Computational Intelligence - February 2016 - 64
Computational Intelligence - February 2016 - 65
Computational Intelligence - February 2016 - 66
Computational Intelligence - February 2016 - 67
Computational Intelligence - February 2016 - 68
Computational Intelligence - February 2016 - 69
Computational Intelligence - February 2016 - 70
Computational Intelligence - February 2016 - 71
Computational Intelligence - February 2016 - 72
Computational Intelligence - February 2016 - Cover3
Computational Intelligence - February 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com