IEEE Computational Intelligence Magazine - November 2019 - 83

http://lipn.univ-paris13.fr/~buscaldi/datasets.zip
11
http://www.romanklinger.de/ironysarcasm/

0.08
0.06
0.04
0.02
0

p(wordIronic)

p(wordSarcastic)

-0.02

lo I
ve
so

This section details different experiments conducted representing the content of tweets by using each of the seven
features models detailed in Section 4.

whereas for the irony/sarcasm detection
task, the number of words was around
15,000 for SEMEVAL and 65,000 for
KLINGER. For both tasks we used the
total number of words in the dictionary
as size of the representation model as we
always noticed a deterioration of the
accuracy when we tried to reduce the
size of the dictionary. As far as the
SEMEVAL dataset is concerned, for the
figurative language dataset, we tried with
the 5,000, 7,000, 15,000 and 55,000
most frequent words, corresponding,
respectively, to more than 5, 3, 2 and
1 occurrences. For the ironic/sarcastic
dataset we tried 1,000, 2,000, 3,000
and 15,000 most frequent words that
correspond, respectively, to more than 5,
3, 2 and 1 occurrences. As far as the
KLINGER dataset is concerned, for the
figurative language dataset, we tried with
the 5,000, 7,000, 15,000 and 70,000
most frequent words, corresponding,
respectively, to more than 11, 8, 3 and
1 occurrences. For the ironic/sarcastic
dataset we tried 5,000, 7,000, 15,000
and 65,000 most frequent words that
correspond, respectively, to more than 9,
6, 3 and 1 occurrences. Tables 1 and 2
show the accuracy and F-measure for
the baseline method in both tasks, using
unigrams according to the bag of word
model (BOW), for the SEMEVAL dataset, different sizes for the representation
model and Naive Bayes classifier. The

in
on
pi
c
bu
ab t
ou
t
th
at
th
ey
w
ho
W
ha ell
pp
ni y
gh
t
da
re y
al
gr ly
ea
t

6. Experiments

The main research questions that we
raised and wanted to answer with these
experiments were: how useful are
semantic features for figurative language
and irony/sarcasm detection? In the
positive case, how much gain are we
able to achieve from within the figurative language detection tasks? As already
mentioned in Section 1, the conducted
experiments are related to the following
two tasks:
❏ detecting tweets using figurative language expressions out of a dataset
which also includes regular tweets;
❏ distinguishing ironic from sarcastic
tweets out of a dataset which includes both.
As described in Section 5, each task
above has been tested on two datasets.
The seven features models were applied
to each dataset. Each tweet was previously processed through a tokenization
step to clean it up by breaking down the
text by spaces and punctuation marks. In
addition to the content representation
strategies, we considered also two different token representations: N-grams and
TF-IDF. To identify a set of useful
n-grams we tokenized the text considering that short forms such as doesn't,
we'll, she's, etc. have to be treated as one
word. For the figurative language detection task, the number of words in the
dictionary was around 55,000 for
SEMEVAL and 70,000 for KLINGER,

to
of

hyperboles ('so...', 'great...', 'really...').
This is in line with the analysis carried
out by [41], which correlated the presence of linguistic markers such as exclamations and intensifiers to sarcasm. Our
curated version of the SEMEVAL dataset (IDs of the tweets and the combinations of unigrams and semantic frames)
is available on-line10.
To verify our results on a different
dataset, we chose the recently published
Irony Sarcasm Analysis Corpus [10], publicly available for download11, which
consists of four sub-corpora: irony, sarcasm, regular and figurative (the last is
ironic and sarcastic, but it has been subsampled to obtain a balanced corpus).
Each original training sub-corpus consisted of 30,000 tweets whereas each test
sub-corpus included 3,000 tweets. The
method of collecting these data is similar
to the one used for the task-11 SemEval
2015, that is, irony and sarcasm hashtags
are considered as self-annotations. As the
dataset contained tweet IDs only, a script
was developed to download each related
tweet content. Because of the same issues
as encountered in the first dataset, since
2016 some tweets have been deleted and,
thus, the size of the sub-corpus is smaller
than the original ones. Each downloaded
training sub-corpus includes around 20k
tweets whereas each downloaded test
sub-corpus includes around 2,400 tweets.
As it can be observed in Figure 5(b),
the terms characterizing the figurative
tweets are indicative of subjectivity, with
a prominence of personal pronouns, like
in the SemEval corpus. In the non-figurative set, the keywords seem more related to factual information, events and the
presence of http links ('via'). We will
refer to the two datasets described above
as the SEMEVAL and the KLINGER
datasets and we have evaluated our
approach on each of them.

Difference

FIGURE 6 Top 10 words for each category in the irony vs. sarcasm dataset, ranked by their relative probability. The leftmost 10 words are the most likely to occur as ironic, the rightmost 10
words are the most likely to occur as sarcastic.

NOVEMBER 2019 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

http://lipn.univ-paris13.fr/~buscaldi/datasets.zip http://www.romanklinger.de/ironysarcasm/

IEEE Computational Intelligence Magazine - November 2019

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2019