Computational Intelligence - February 2015 - 29

Cost

false positive rate and the
Table 3 Configuration and cost of the five ensemble strategies deployed on Amazon EC2 with FCUBE.
true positive rate of all the
learner
rule lisT rule Tree GPF
MPlCs
sbbJ
ToTal
classifiers generated with
Factoring
Data sample
1%
1%
1%
1%
1%
-
the five lear ners. Two
conFiguration rate
important observations are
Variable
100%
100%
100%
100%
100%
-
worth commenting. First,
sample rate
False negatiVe 0.47
0.47
0.47
-
-
-
MPLCS and SBBJ generate
weight
classifiers covering a wide
ensemble
buDget
$20
$20
$20
$20
$20
$100
section of the TPR/FPR
conFiguration
trade-off. On the other
instances
100
100
100
100
100
500
FlaVor
c3.xlarge c3.xlarge c3.xlarge m3.meDium m3.meDium -
hand, RL, RT, and GPF are,
Virtual cpus
4
4
4
1
1
-
broadly speaking, specialcost per hour $0.210
$0.210
$0.210
$0.070
$0.070
-
ized in some region of the
time (min)
57
57
57
171
171
-
front. This is explained by
Virtual cpu
1800
4 # 100
4 # 100
4 # 100
3 # 100
3 # 100
the fact that RL, RT, and
hours
GPF receive the cost of the
two errors as parameters of
classifiers exhibiting a cost higher than the baseline with
the learning process. As a result, the models generated by these
respect to the fusion set D f have been discarded during the filalgorithms are concentrated in the region of the trade-off that
minimizes the established cost. Second, it is interesting to see that
tering process. We can see that all the learners improve on the
all the regions of the trade-off are covered by one or another
baseline cost. GP Function learner generates classifiers that outlearner. The Rule Tree learner is however clearly dominated by
perform those generated with Rule Tree and Rule List. SBBJ
the other learners, and thus is not likely to contribute to build a
and MPLCS generate classifiers that cover a wide range of valcollaborative solution. This analysis confirms that EC-based
ues, it is therefore hard to compare them against the other
learners generate classifiers that exhibit a remarkable diversity.
learners. The next step of the analysis consists in determining
To compare the performance of the different learners, we
whether the observed diversity helps building ensembles that
analyze the cost (Bayesian Risk) of the individual classifiers as
will improve the accuracy of the independent classifiers.
in (2). The weight of the two errors is set according to the class
balance of the problem with the goal of penalizing classifiers
that do not perform well on the minority class. Thus, in this
0.55
case, we set the weights of the false positive and false negative
rates to 0.53 and 0.47 respectively. This analysis is useful to
0.50
determine, for the considered problem, what are the most
0.45
appropriate learners to use. Fig. 8a shows the testing set cost for
0.40
the five different learners. Additionally, we plot the cost of a
0.35
naive classifier that always predicts the majority class. Note that
0.30
0.25
1

Rule
List

Rule
Tree

GP
MPLCS
Function

SBBJ

(a)
0.55
0.50

0.6
Rule List
Rule Tree
GP Funtion
MPLCS
SBBJ

Majority
Vote

0.35
0.30

0.4
0.6
False Positive Rate

0.8

FiGure 7 trade-off between false positive rate and true positive rate
of the 500 classifiers evaluated on the test set. we train 100 classifiers with each of the five learners integrated in the framework.

00
L5
AL

-1
00
BJ

S-

10
0
SB

0.2

M
PL
C

PF
-1
00

RT
-1
00

0.25

G
0

Logistic
Regression

0.40

0.2

Best
of Run

0.45

L10
0

0.4

Cost

True Positive Rate

0.8

(b)
FiGure 8 cost or bayesian risk of (a) individual classifiers and (b)
best model of the run and fused models obtained with majority Vote
and logistic regression. in all cases lower is better.

february 2015 | Ieee ComputatIonal IntellIgenCe magazIne

Table of Contents for the Digital Edition of Computational Intelligence - February 2015