Computational Intelligence - February 2016 - 42

Because classification and regression have
similarities and differences, there are common and specific ensemble methods for
classification and regression. It is therefore
more suitable to discuss the ensemble methods for classification and regression together.
The paper is organized as follows: in
Section II, we review the theories of
ensemble methods; Ensemble methods
for classification and regression are presented in Section III; In Section IV, we
summarize the paper. We suggest future
research directions in Section V.

Table 1 Nomenclature.
abbReVIaTION

DeFINITION

AdaBoost

ADAPTIVE BOOSTING

ANFIS

ADAPTIVE NEURO-FUZZY INFERENCE SYSTEM

ANN

ARTIFICIAL NEURAL NETWORK

ARCING

ADAPTIVELY RESAMPLE AND COMBINE

ARIMA

AUTO-REGRESSIVE INTEGRATED MOVING AVERAGE

ARTMAP

PREDICTIVE ADAPTIVE RESONANCE THEORY

Bagging

BOOTSTRAP AGGREGATION

CNN

CONVOLUTIONAL NEURAL NETWORK

DENFIS

DYNAMIC EVOLVING NEURAL-FUZZY INFERENCE SYSTEM

DIVACE

DIVERSE AND ACCURATE ENSEMBLE LEARNING ALGORITHM

II. Theory

DNN

DEEP NEURAL NETWORK

The main theory behind ensemble methods is bias-variance-covariance decomposition. It offers theoretical justification for
improved performance of an ensemble
over its constituent base predictors. The
key to ensemble methods is diversity,
which includes data diversity, parameter
diversity, structural diversity, multi-objective optimization and fuzzy methods.

EMD

EMPIRICAL MODE DECOMPOSITION

GLM

GENERALIZED LINEAR MODELS

IMF

INTRINSIC MODE FUNCTION

KNN

K NEAREST NEIGHBOR

LR

LINEAR REGRESSION

LS-SVR

LEAST SQUARE SUPPORT VECTOR REGRESSION

MKL

MULTIPLE KERNEL LEARNING

MLMKL

MULTI-LAYER MULTIPLE KERNEL LEARNING

MLP

MULTIPLE LAYER PERCEPTRON

A. Bias-Variance-Covariance
Decomposition

MPANN

MEMETIC PARETO ARTIFICIAL NEURAL NETWORK

MPSVM

MULTI-SURFACE PROXIMAL SUPPORT VECTOR MACHINE

Researchers initially investigated the theory behind ensemble methods by using
regression problems. In the context of
regression, there is a theoretical proof to
show that a proper ensemble predictor can
guarantee to have smaller squared error
than the average squared error of the base
predictors.The proof is based on ambiguity decomposition [11], [12]. However,
ambiguity decomposition only applies to a
single dataset with ensemble methods. For
multiple datasets, bias-variance-covariance
decomposition is introduced [12]-[15]
and the equation is shown:

NCL

NEGATIVE CORRELATION LEARNING

PCA

PRINCIPAL COMPONENT ANALYSIS

PSO

PARTICLE SWARM OPTIMIZATION

QCQP

QUADRATICALLY CONSTRAINED QUADRATIC PROGRAM

SMO

SEQUENTIAL MINIMAL OPTIMIZATION

SVM

SUPPORT VECTOR MACHINE

SVR

SUPPORT VECTOR REGRESSION

RBFNN

RADIAL BASIS FUNCTION NEURAL NETWORK

RNN

RECURRENT NEURAL NETWORK

RT

REGRESSION TREE

RVFL

RANDOM VECTOR FUNCTIONAL LINK

E [ f - t] 2 = bias 2 + 1 var
M
+ c 1 - 1 m covar
M
1
bias =
/ ^E [fi] - t h
M i
var = 1
M

/ E [fi - E [fi]] 2
i

1
//
M (M - 1) i j ! i
E [fi - E [fi]] (f j - E [f j])

covar =

(3)

where t is the target and fi is the output from each model and M is the size

42

of ensemble. The error is composed of
the average bias (which measures the
average difference between the prediction of the base learner and the desired
output), plus a term involving their
average variance (which measures the
average variability of the base learners),
and the third term involving their average pairwise covariance term (which
measures the average pairwise difference
of different base learners).
There exist several theoretical
insights about the soundness of using
ensemble methods such as strength-correlation [3], stochastic discrimination

IEEE ComputatIonal IntEllIgEnCE magazInE | FEbruary 2016

[16] and margin theory [17]. They have
been shown to be equivalent to biasvariance-covariance decomposition [18].
From the equation, we can see the
term covar can be negative, which may
decrease the expected loss of the
ensemble while leaving bias and var
unchanged. Beside the covar, the number
of models also plays an important role.
As it increases, the proportion of the
variance in the overall loss vanishes
whereas the impor tance of the
covar iance increases. Overall, this
decomposition shows that if we are able
to design lowcorrelated individual



Table of Contents for the Digital Edition of Computational Intelligence - February 2016

Computational Intelligence - February 2016 - Cover1
Computational Intelligence - February 2016 - Cover2
Computational Intelligence - February 2016 - 1
Computational Intelligence - February 2016 - 2
Computational Intelligence - February 2016 - 3
Computational Intelligence - February 2016 - 4
Computational Intelligence - February 2016 - 5
Computational Intelligence - February 2016 - 6
Computational Intelligence - February 2016 - 7
Computational Intelligence - February 2016 - 8
Computational Intelligence - February 2016 - 9
Computational Intelligence - February 2016 - 10
Computational Intelligence - February 2016 - 11
Computational Intelligence - February 2016 - 12
Computational Intelligence - February 2016 - 13
Computational Intelligence - February 2016 - 14
Computational Intelligence - February 2016 - 15
Computational Intelligence - February 2016 - 16
Computational Intelligence - February 2016 - 17
Computational Intelligence - February 2016 - 18
Computational Intelligence - February 2016 - 19
Computational Intelligence - February 2016 - 20
Computational Intelligence - February 2016 - 21
Computational Intelligence - February 2016 - 22
Computational Intelligence - February 2016 - 23
Computational Intelligence - February 2016 - 24
Computational Intelligence - February 2016 - 25
Computational Intelligence - February 2016 - 26
Computational Intelligence - February 2016 - 27
Computational Intelligence - February 2016 - 28
Computational Intelligence - February 2016 - 29
Computational Intelligence - February 2016 - 30
Computational Intelligence - February 2016 - 31
Computational Intelligence - February 2016 - 32
Computational Intelligence - February 2016 - 33
Computational Intelligence - February 2016 - 34
Computational Intelligence - February 2016 - 35
Computational Intelligence - February 2016 - 36
Computational Intelligence - February 2016 - 37
Computational Intelligence - February 2016 - 38
Computational Intelligence - February 2016 - 39
Computational Intelligence - February 2016 - 40
Computational Intelligence - February 2016 - 41
Computational Intelligence - February 2016 - 42
Computational Intelligence - February 2016 - 43
Computational Intelligence - February 2016 - 44
Computational Intelligence - February 2016 - 45
Computational Intelligence - February 2016 - 46
Computational Intelligence - February 2016 - 47
Computational Intelligence - February 2016 - 48
Computational Intelligence - February 2016 - 49
Computational Intelligence - February 2016 - 50
Computational Intelligence - February 2016 - 51
Computational Intelligence - February 2016 - 52
Computational Intelligence - February 2016 - 53
Computational Intelligence - February 2016 - 54
Computational Intelligence - February 2016 - 55
Computational Intelligence - February 2016 - 56
Computational Intelligence - February 2016 - 57
Computational Intelligence - February 2016 - 58
Computational Intelligence - February 2016 - 59
Computational Intelligence - February 2016 - 60
Computational Intelligence - February 2016 - 61
Computational Intelligence - February 2016 - 62
Computational Intelligence - February 2016 - 63
Computational Intelligence - February 2016 - 64
Computational Intelligence - February 2016 - 65
Computational Intelligence - February 2016 - 66
Computational Intelligence - February 2016 - 67
Computational Intelligence - February 2016 - 68
Computational Intelligence - February 2016 - 69
Computational Intelligence - February 2016 - 70
Computational Intelligence - February 2016 - 71
Computational Intelligence - February 2016 - 72
Computational Intelligence - February 2016 - Cover3
Computational Intelligence - February 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com