IEEE Computational Intelligence Magazine - November 2019 - 84

experiment also served to justify the use
of the full vocabulary size for the two
different datasets. It is not surprising that
the highest value for the vocabulary size
produces the best results for both tasks,
given the small size of tweets. Moreover,
some hashtags, one of the distinctive fea-

tures of tweets, may occur rarely but
they are highly informative keywords.
Similarly, for KLINGER, Tables 3
and 4 indicate the accuracy and F-measure
we obtained for the baseline method in
both tasks. In particular, for the first task,
we used the figurative (balanced) and

TABLE 1 Setting the threshold for the optimal feature size for the figurative
language dataset of SEMEVAL and baseline accuracy and F-measure comparison
using unigrams.
BOW - 5,000
(> 5 OCCURRENCES)

BOW - 7,000
(> 3 OCCURRENCES)

BOW - 15,000
(> 2 OCCURRENCES)

BOW - 55,000
(> 1 OCCURRENCE)

ACC.

0.85

0.86

0.87

0.90

F.

0.86

0.89

0.90

0.92

TABLE 2 Setting the threshold for the optimal feature size for the irony/sarcasm
dataset of SEMEVAL and baseline accuracy and F-measure comparison using
unigrams.
BOW - 1,000
(> 5 OCCURRENCES)

BOW - 2,000
(> 3 OCCURRENCES)

BOW - 3,000
(> 2 OCCURRENCES)

BOW - 15,000
(> 1 OCCURRENCE)

ACC.

0.84

0.86

0.82

0.88

F.

0.88

0.89

0.85

0.93

TABLE 3 Setting the threshold for the optimal feature size for the figurative
language dataset of KLINGER and baseline accuracy and F-measure comparison
using unigrams.
BOW - 5,000
BOW - 7,000
BOW - 15,000
BOW - 70,000
(> 11 OCCURRENCES) (> 8 OCCURRENCES) (> 3 OCCURRENCES) (> 1 OCCURRENCE)
ACC. 0.80

0.85

0.87

0.89

F.

0.82

0.86

0.88

0.79

TABLE 4 Setting the threshold for the optimal feature size for the irony/sarcasm
dataset of KLINGER and baseline accuracy and F-measure comparison using
unigrams.
BOW - 5,000
(> 9 OCCURRENCES)

BOW - 7,000
(> 6 OCCURRENCES)

BOW - 15,000
(> 3 OCCURRENCES)

BOW - 65,000
(> 1 OCCURRENCE)

ACC.

0.75

0.76

0.77

0.78

F.

0.76

0.77

0.78

0.79

TABLE 5 Accuracy and F-measure for the figurative language detection problem
across the different features models on SEMEVAL.

84

FEATURES

ACC. TF.IDF

ACC. BOW

F. TF.IDF

F. BOW

UNIGRAMS

0.72

0.90

0.74

0.92

BABELNET SYNSETS

0.73

0.86

0.74

0.88

SEMANTIC FRAMES

0.56

0.68

0.59

0.72

BNS+SF

0.72

0.86

0.75

0.88

UNIGRAMS+BNS

0.77

0.94

0.79

0.95

UNIGRAMS+SF

0.76

0.92

0.78

0.93

UNIGRAMS+BNS+SF

0.74

0.92

0.75

0.94

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2019

the regular dataset whereas for the second task we used the irony and the sarcasm datasets. To note that irony and
sarcasm tweets size of KLINGER are
much higher than those of SEMEVAL;
this explains why we had higher threshold values in Table 4.
For each dataset, two Naive Bayes
classifiers were trained using unigram
features and TF-IDF features. In the second case, each word was represented
using its TF-IDF value instead of frequency. Tables 5 and 6 show the accuracy and F-measure obtained for each of
the seven features models in the figurative language detection task for
SEMEVAL whereas Tables 7 and 8
show results on KLINGER. The results
have been calculated using 10-fold
cross-validation and averaged over 10
runs (also for the baselines/unigrams).
The accuracy has been calculated as
accuracy = (TP + TN )/(TP + TN + FP +
FN ) and the F-measure as: F-measure =
(2 ) TP )/(2 ) TP + FP + FN ), where TP,
TN, FP and FN correspond, respectively,
to true positives, true negatives, false
positives and false negatives. Table 9 and
Table 10 illustrate the process of computing TP, TN, FP and FN with the help
of the confusion matrices for the two
tasks with values for the baselines on
SEMEVAL. It should be noticed that for
the task 1 and 2, an under-sampling of
the majority class (sarcastic and nonfigurative tweets) was performed. Sarcastic tweets were reduced from 2,081 to
1,350 whereas non-figurative tweets
were reduced from 8,959 and to 3,400.
We thus obtained balanced datasets for
the two classification tasks.
For task 1, using semantic features
improves the classification with respect to
unigrams (which represents our baseline)
for all the combinations including both
unigrams and the semantic features. The
combinations that resulted in the lowest
accuracy (lower than the baseline) were
those employing semantic frames without
unigrams (e.g., BabelNet synsets, semantic frames, BabelNet synsets + semantic
frames). This suggests a connection
between the style of text when adopting
figurative language expression to denote
either irony or sarcasm. Also, the adoption
IEEE Computational Intelligence Magazine - November 2019

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2019
