Computational Intelligence - February 2014 - 29

❏ Topic and opinion models index, which deals with the indexing

of the discovered/detected topic associated with their sentiment models.
❏ Fuzzy reasoning, which will be utilized to handle multiple
opinion feature scores in a more flexible (natural, humanfriendly) manner. This will be done by using fuzzy varieties
of OWL, such as Fuzzy OWL2 [3], and the related automated reasoners;
❏ Learning mechanisms, sentic pattern discovery, and combinatorial
optimization, which will be investigated to further improve
the accuracy of the proposed approach. In our case, the
graph representations of FRED, and its augmentation with
the Sentilo model, provide ordered sets of features that will
be reused to train machine learning classifiers, and to perform sentiment patterns discovery, as a variety of knowledge
patterns [19]. Another interesting direction is in extending
combinatorial optimization applied to parsing trees (as e.g.
reported in [26]) to FRED's semantic graphs;
❏ Analogical reasoning, which can be used to deal with sarcasm
by extending the known or newly discovered sentic patterns to figurative and ironic modalities. Here AffectiveSpace [8] can play an important role as well, once its
concepts are associated to frame-based knowledge.
5. Results and Evaluation

Sentilo has been evaluated on 50 sentences, including a subset
of the MPQA opinion corpus [11], and sentences extracted
from the Europarl corpus10. MPQA was originally used for
the opinion analysis task with training and test cases. Our
model does not require training, since it is based on the
semantic output of FRED. We can then compare our
approach to the original annotations about opinion holders11
provided in the corpus, incremented with appropriate annotations made by three annotators also for the main topics and
subtopics of those opinions.
Three annotators have been instructed on our assumptions
about events and situations to be taken as preferential main
topics, and once they have expressed their choices, they have
been allowed to discuss in cases of disagreement.
They have reached a complete agreement in all cases for
holder's identification (modulo the three amendments made to
the original benchmark), in 47 cases out of 50 for topic identification (.96 is the average interrater proportional agreement),
and in 36 cases out of 50 for subtopic identification (.81 is the
average interrater proportional agreement).12 When annotators
did not reach a complete agreement, we accepted the choice of
the majority (2 out of 3).
10

http://www.statmt.org/europarl/
Our three annotators actually amended three holder annotations, which resulted inaccurate.
12
We have used proportional agreement because in this case we need an absolute agreement measure, not a relative reliability measure such as Cohen's k. Reliability measures
are important when variation within the sample is of interest, e.g. when the research
question refers to suitability of a test in a particular setting, cf. [15], which is not the
case of simple acceptability of annotations. In such simple cases reliability can even
produce paradoxical results, as reported by [15].
11

Table 1 Information measures for holder and topic detection.
Measure/
Task

Holder
deTecTion

Topic
deTecTion

subTopic
deTecTion

Precision

0.99

0.72

0.77

recall

0.91

0.64

0.77

0.95

0.68

0.77

accuracy

0.95

0.66

0.80

The information measures for holder detection and topic
detection (Table 1) include precision, recall, F1, and accuracy,
where the positives and negatives have been calculated locally
on each sentence output, i.e. the maximum score for each sentence is 1 for true positives plus false negatives, and for false
positives plus true negatives. The reason for this localization is
straightforward: each sentence is expected to provide a closed
set of holders and topics, and we are interested in the performance of Sentilo for each sentence, not for the joint opinion
space of the collection of sentences. The results are very satisfactory for holder detection, where two heuristics provide very
high accuracy for most cases, i.e. choosing agentive entities in
triggering events, or choosing an implicit holder when all
explicit heuristics fail.
The results are good also for main topic detection, a novel
task in the way we perform it here, i.e. by assuming a Davidsonian world, where main entities are assumed to be events or situations, except when purely attributive states of a affairs are referenced. Consider that for this first test, we have applied a few
heuristical rules for subgraph mining, without refining them,
and without overfitting them after evaluation. Our intent is
actually to have a feedback on the validity of our hypotheses,
while leaving the optimization of detection methods, and their
application to the ranking of opinion features.
As expected, difficult sentences for main topic detection
(14/50) seem to concentrate on unusual syntactic constructions within complex sentences, which require specific rules
to enable topic detection. Difficult sentences for topic detection often have an implicit holder (6/14 while the expected
distribution should be 4/14), which may indicate an insufficiency in the subgraph extraction heuristics currently used
for sentences with implicit holders.
For subtopics, the results are slightly better than main topics, probably because of the strong connectedness of FRED
graphs that allows reaching at subtopics even from wrong
main topics. This happens because FRED has been designed
to maximize the amount of links between the semantic elements expressed by a sentence: syntactic dependencies, compositional lexical dependencies, automatic typing, robust
semantic role labeling even in the absence of a known frame,
disambiguation of named entities and word senses. Therefore,
the (subjective at the moment) impression is that even when
the primary topic attachment is not expected, a correct subtopic can be still reached through one of the paths activated
by FRED. Heuristics refinement is however barely due in this

February 2014 | Ieee ComputatIonal IntellIgenCe magazIne

Table of Contents for the Digital Edition of Computational Intelligence - February 2014