IEEE Computational Intelligence Magazine - May 2018 - 25

TABlE 3 Average rankings of the methods computed
with Friedman Aligned Ranks (FAR) and Holm's adjusted
p-values (pHolm).
Acc
METhoD

FAr

pholm

FAr

pholm

MC ESVM

50.28

51.20

oVo

70.36

0.1611

75.30

0.1128

MSVM-CS

83.82

0.0385

78.54

0.1128

oVA

92.78

0.0091

94.56

0.0074

MSVM2

103.44

0.0008

102.94

0.0015

MSVM-ww

105.04

0.0007

101.12

0.0020

MSVM-llw

110.28

0.0002

112.34

0.0001

TABlE 4 Average Friedman Aligned Ranks (FAR) and Holm's
adjusted p-values (pholm) for the best methods.
Acc

METhoD

FAr

pholm

FAr

pholm

MC2ESVM

27.16

27.96

oVo

39.92

0.0385

41.74

0.0254

MSVM-CS

46.92

0.0027

44.33

0.0161

1.0
0.8
Probability

classes, respectively. On the other hand, its best performance
is shown in the Cleveland, Ecoli, and Zoo datasets, which
have at least five classes each.
❏ It is worth noting that, in general, MC2ESVM significantly
performs better than reference methods in datasets with
imbalance rates greater than 1.5 and with five or more class-
es. This may be explained due to the fact that OVO with a
large number of classes, significantly increases the number of
binary classifiers, leading to an ensemble with a more com-
plex decision function. OVA, on the other hand, artificially
makes higher this imbalance.
❏ On well-balanced problems, the performance of MC2ES-
VM and reference methods are quite similar, regardless of
the number of classes.
❏ MC2ESVM statistically outperforms most of the SVM for-
mulations for handling multiclass problems. In fact, it is sta-
tistically better in five out of six methods for the accuracy
score and in four out of six methods for Cohen's Kappa sta-
tistic, under the considered level of a = 0.05.
❏ MSVM-CS and the OVO decomposition are clearly the
most competitive multiclass SVMs for the proposed
MC2ESVM. These competitive performances can also be
noted in the lack of a statistically significant difference when
Cohen's Kappa is considered.
❏ The difference between MC2ESVM and OVO in accuracy
is marginal, but in Cohen's Kappa, MC2ESVM clearly out-
performs OVO. This is an interesting point to observe
because the hyper-parameters for each method were done
by considering the accuracy as the main criterion. Thus,
MC2ESVM is not overfitted to this criterion.
❏ Except for MSVM-CS, the rest of the multiclass SVMs
based on modifying the objective function showed a low
performance. This may be due to the fact that they have to
deal with a larger optimization problem than those based
on decomposition.
❏ OVO can be highlighted as the best method based on
decomposing the problem into multiple binary classification
problems, while MSVM-CS is an outstanding method from
those based on modifying the optimization problem.
The Holm's test has reported no statistically significant dif-
ference between MC 2 ESVM and OVO neither between
MC2ESVM and MSVM-CS, when the multiple comparison is
done by considering all methods. This may be due to the num-
ber of algorithms in the comparison and the fact that those
algorithms have influence on the rank computation and also in
the post-hoc [39]. Therefore, we have thoroughly inspected
these three algorithms by comparing them. With this aim,
Table 4 shows the statistical comparison when considering
these methods. MC 2ESVM is again considered as a control
method in the test. Based on this more focused test, we can
note that indeed the differences between MC2ESVM and the
reference methods (OVO and MSVM-CS) are statistically sig-
nificant at the considered level.
Fig. 3 graphically depicts a comparison of the computation-
al time for each method. This figure represents the probability

0.6
0.4
0.2
0.0

MC2ESVM
OVA
OVO
MSVM-WW
MSVM-LLW
MSVM-CS
MSVM2
0 50 100 150 200 250 300 350 400 450 500 550 600
Times (s)

FigurE 3 Probability of each method to learn a Multiclass SVM classifier in a given amount of time.

that a given method learns the multiclass SVM in a given
amount of time. From it, we can note:
❏ Both OVO and OVA are the best ones, and have virtually
the same performance.
❏ MC 2ESVM is the second best one, requiring at most,
30 seconds for solving each benchmark problem.
❏ Single machine methods are clearly the slowest ones.
This is due to the fact that they deal with a larger optimiza-
tion problem.
❏ In the best case, single machine methods required around
280 seconds to ensure solving each dataset.
❏ Among all SVMs multiclass extensions, MSVM-CS is the
worst one in terms of computational time.

may 2018 | IEEE ComputatIonal IntEllIgEnCE magazInE

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2018