Computational Intelligence - November 2017 - 65

mapping is employed) without explicitly
knowing the form of the feature transformation function z (x).
Solving Eqn. ( 7 ), we can get
w = R iN= 1 a i z (x i) . Hence, the discriminant function can be written as:
g^xih =

N

/ a j y j k^x j, x ih + b.

(8)

j=1

C. Kernel Ridge Regression

KRR [14], [15] (and Section 12.3.7 in
[16]) extends the ridge regression to
non-linear cases via the kernel trick. It
was originally proposed to solve the
regression problem and is shown to
achieve similar performance to more
sophisticated models such as the support
vector regression. However, the classification ability of the KRR has been
under-researched [15]. In this work, we
conduct a comprehensive study on
KRR for classification and propose a
novel KRR ensemble method.
A typical linear regression problem
can be formulated as:
min
/ ^w < x i - y ih2 + C w 2,
w
2

(9)

i

where parameter C is a user-defined
regularization parameter that controls
the model complexity. This problem can
lead to an elegant closed-form solution:
w = ^X < X + C I h-1 X < Y,

(10)

where the data matrix X has one sample per row x i, and each element of the
vector Y is the output target y i of x i.
I is an identity matrix.
KRR extends the linear regression
nonlinearly through the kernel trick.
Based on the Representer theorem [19],
the solution of w can be formulated as a
linear combination of the samples in the
feature space z (x) as: w = R i a i z (x i).
The KRR problem is then formulated as:
2

min
Y - Ka + Ca < Ka.
a

(11)

Similarly, the solution is given in a
closed-form manner as:
a

= (K + C I) -1 Y.

(12)

In the same way, by applying
the kernel trick, the kernel matrix K
can be obtained by K ij = k (x i, x j) =
T
z (x i) z (x j).
A classification problem can be posed
as a regression problem by defining the
output target Y with 0 - 1 encoding
[15]. More specifically, if there are L
classes with N samples, the output Y
should be an N # L matrix which can
be generated by the following equation:
1 If ith sample belongs to
. (13)
Yij = * the jth class
0 Otherwise
D. Feedforward Neural Network

Single hidden layer feedforward neural
network (SLFN) gains its popularity in
classification amongst the family of
feedforward neural networks because of
its global approximation ability. Fig. 2
demonstrates the basic structure of an
SLFN which consists of three layers:
input layer, hidden layer and output
layer. Denote the input data samples as
X and their corresponding output classes as Y. The input features are firstly
linearly scaled by the weights a between
the input and hidden layer. After that, a
nonlinear activation function U h is
applied to the transformed features to
get the features in the hidden layer.
H = U h ^Xa + bh .

(14)

The bias vector b in Eqn. (14) can
be omitted by augmenting the input as
x = [x <, 1] < and Eqn. (14) can be simplified as:
H = U h ^Xa h .

(15)

The features H are forwarded to the
output layer. The output layer computes
the loss of the data samples by comparu from the SLFN and
ing the output Y
the ground truth label Y:
u = U o ^Hb h, d = l ^ Y, Y
u h.
Y

β

a

Input
Layer

Hidden
Layer

Output
Layer

FIGURE 2 The structure of SLFN.

of loss function can be problem dependent and common choices of loss function are hinge loss, mean square loss,
logistic loss and so on.
E. Boosting

Most boosting algorithms work by iteratively training unstable classifiers with
respect to a distribution and adding them
to a final stable classifier. Boosting methods operate in a "divide and conquer"
manner. When a new base classifier is
generated, misclassified examples gain
weight while correctly-classified examples lose weight. Thus, future unstable
learners focus more on the examples that
previous models misclassified. Due to
page limit, we refer the interested readers
to [7] for more information.
F. Rotation Forest

Rotation Forest (RoF) is also a wellknown DT ensemble. It constructs "high
strength" and "low correlation" classifiers [20]. RoF differs from RaF mainly
in two aspects: firstly, RoF applies feature extraction based on a rotation
matrix for each DT. Secondly, all features are used as candidate features when
searching for the "optimal feature" for a
hyperplane in RoF while random subspace is used in RaF.

(16)

u is obtained by
Usually the output Y
transforming the features H from the
hidden layer directly without any nonlinear activation function. In this case,
U o (H b) = Hb. Moreover, the choices

IV. Oblique DT Ensemble

Orthogonal DT learning algorithms
work in a greedy top down fashion. For
each node, it evaluates the score function exhaustively for each candidate feature and all possible thresholds for that

NOVEMBER 2017 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

65



Table of Contents for the Digital Edition of Computational Intelligence - November 2017

Computational Intelligence - November 2017 - Cover1
Computational Intelligence - November 2017 - Cover2
Computational Intelligence - November 2017 - 1
Computational Intelligence - November 2017 - 2
Computational Intelligence - November 2017 - 3
Computational Intelligence - November 2017 - 4
Computational Intelligence - November 2017 - 5
Computational Intelligence - November 2017 - 6
Computational Intelligence - November 2017 - 7
Computational Intelligence - November 2017 - 8
Computational Intelligence - November 2017 - 9
Computational Intelligence - November 2017 - 10
Computational Intelligence - November 2017 - 11
Computational Intelligence - November 2017 - 12
Computational Intelligence - November 2017 - 13
Computational Intelligence - November 2017 - 14
Computational Intelligence - November 2017 - 15
Computational Intelligence - November 2017 - 16
Computational Intelligence - November 2017 - 17
Computational Intelligence - November 2017 - 18
Computational Intelligence - November 2017 - 19
Computational Intelligence - November 2017 - 20
Computational Intelligence - November 2017 - 21
Computational Intelligence - November 2017 - 22
Computational Intelligence - November 2017 - 23
Computational Intelligence - November 2017 - 24
Computational Intelligence - November 2017 - 25
Computational Intelligence - November 2017 - 26
Computational Intelligence - November 2017 - 27
Computational Intelligence - November 2017 - 28
Computational Intelligence - November 2017 - 29
Computational Intelligence - November 2017 - 30
Computational Intelligence - November 2017 - 31
Computational Intelligence - November 2017 - 32
Computational Intelligence - November 2017 - 33
Computational Intelligence - November 2017 - 34
Computational Intelligence - November 2017 - 35
Computational Intelligence - November 2017 - 36
Computational Intelligence - November 2017 - 37
Computational Intelligence - November 2017 - 38
Computational Intelligence - November 2017 - 39
Computational Intelligence - November 2017 - 40
Computational Intelligence - November 2017 - 41
Computational Intelligence - November 2017 - 42
Computational Intelligence - November 2017 - 43
Computational Intelligence - November 2017 - 44
Computational Intelligence - November 2017 - 45
Computational Intelligence - November 2017 - 46
Computational Intelligence - November 2017 - 47
Computational Intelligence - November 2017 - 48
Computational Intelligence - November 2017 - 49
Computational Intelligence - November 2017 - 50
Computational Intelligence - November 2017 - 51
Computational Intelligence - November 2017 - 52
Computational Intelligence - November 2017 - 53
Computational Intelligence - November 2017 - 54
Computational Intelligence - November 2017 - 55
Computational Intelligence - November 2017 - 56
Computational Intelligence - November 2017 - 57
Computational Intelligence - November 2017 - 58
Computational Intelligence - November 2017 - 59
Computational Intelligence - November 2017 - 60
Computational Intelligence - November 2017 - 61
Computational Intelligence - November 2017 - 62
Computational Intelligence - November 2017 - 63
Computational Intelligence - November 2017 - 64
Computational Intelligence - November 2017 - 65
Computational Intelligence - November 2017 - 66
Computational Intelligence - November 2017 - 67
Computational Intelligence - November 2017 - 68
Computational Intelligence - November 2017 - 69
Computational Intelligence - November 2017 - 70
Computational Intelligence - November 2017 - 71
Computational Intelligence - November 2017 - 72
Computational Intelligence - November 2017 - 73
Computational Intelligence - November 2017 - 74
Computational Intelligence - November 2017 - 75
Computational Intelligence - November 2017 - 76
Computational Intelligence - November 2017 - 77
Computational Intelligence - November 2017 - 78
Computational Intelligence - November 2017 - 79
Computational Intelligence - November 2017 - 80
Computational Intelligence - November 2017 - 81
Computational Intelligence - November 2017 - 82
Computational Intelligence - November 2017 - 83
Computational Intelligence - November 2017 - 84
Computational Intelligence - November 2017 - 85
Computational Intelligence - November 2017 - 86
Computational Intelligence - November 2017 - 87
Computational Intelligence - November 2017 - 88
Computational Intelligence - November 2017 - 89
Computational Intelligence - November 2017 - 90
Computational Intelligence - November 2017 - 91
Computational Intelligence - November 2017 - 92
Computational Intelligence - November 2017 - 93
Computational Intelligence - November 2017 - 94
Computational Intelligence - November 2017 - 95
Computational Intelligence - November 2017 - 96
Computational Intelligence - November 2017 - 97
Computational Intelligence - November 2017 - 98
Computational Intelligence - November 2017 - 99
Computational Intelligence - November 2017 - 100
Computational Intelligence - November 2017 - 101
Computational Intelligence - November 2017 - 102
Computational Intelligence - November 2017 - 103
Computational Intelligence - November 2017 - 104
Computational Intelligence - November 2017 - Cover3
Computational Intelligence - November 2017 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com