Computational Intelligence - August 2016 - 43

1.0

Accuracy
MAE
RMSE

0.9

Accuracy
MAE
RMSE

0.8
0.7

0.8
0.6

0.7
0.6

0.5

0.5
0.4

0.4
0.3

Neural

Linear

SVM

0.3

RNN

LSTM

GRU

Figure 5 Experimental results of traditional classifiers vs. a pure
-neural framework on the Yelp 2013 dataset.

Figure 6 Experimental results of different sequence models on the
Yelp 2013 dataset.

shows that the temporal relations indeed characterize users and
products better.

5. Conclusion

4.3.3 Comparison of Traditional Classifiers with a Pure
Neural Framework
There are two main steps in our approach: firstly, learning user,
product and review embeddings; secondly, feeding these
embeddings into SVM training. We also design a pure neural
version of our approach, in which SVM is replaced by a fully
connected layer with the softmax function and hence sentiment classification can be performed in the process of representation learning.
We present the experimental results of this pure neural
framework and our approach using four different classifiers in
Fig. 5, where Neural indicates this pure neural framework.
LR, NB and Linear indicate our approach with SVM replaced
by logistic regression [69], Naive Bayes, and LIBLINEAR
[70], respectively.
It is observed that neural framework outperforms logistic
regression and Naive Bayes, but gives worse results compared
to LIBLINEAR and SVM on all the three metrics. The best
performance is obtained using SVM, which outperforms the
neural framework by a large margin. This shows that for the
datasets experimented here, it is better to separate representation learning from sentiment classifier training.
4.3.4 Comparison with Different Sequence Models
We have also experimented with different sequence models,
including RNN and LSTM, for learning user and product representations. Fig. 6 shows the results of using RNN, LSTM
and GRU as a sequence model on the Yelp 2013 dataset. It can
be observed that GRU outperforms LSTM, which in turn
gives better performance compared to RNN on all the three
metrics. It is because both GRU and LSTM, having more
complicated hidden units, offer better composition capability
than RNN. GRU has fewer parameters to train compared to
LSTM and thus generalizes better than LSTM.

This paper has presented a sequence modeling based neural
network approach for document-level sentiment analysis. The
approach employs RNN-GRU to learn user and product
embeddings from the temporal ordered review documents.
These embeddings, together with review embeddings learned
by a CNN, are used to train SVMs for sentiment classification.
We have conducted extensive experiments on three review
datasets using three evaluation metrics. Empirical results show
that our approach achieves the state-of-the-art performance on
all these datasets. We have found that (1) modeling reviews as
sequences rather than unordered sets boost the performance of
user and product representation composition; (2) concatenating
review, user and product embeddings for training SVMs for
sentiment classification gives superior results compared to a
pure neural framework and beats the best results reported so far.
Evaluations on three large-scale datasets show that the proposed
method performs better than several strong baseline methods
which regard reviews as unordered set.
In future work, we plan to explore other sequence learning
model, such as bidirectional RNN, bidirectional LSTM and gated
feedback RNN for sentiment analysis. We will also explore other
methods in learning user and product embeddings and investigate
the feasibility of using these embeddings for a wide range of tasks
such as product recommendation and product sales prediction.
Acknowledgments

This work is supported by the National Natural Science Foundation of China 61370165, National 863 Program of China
2015AA015405, Shenzhen Development and Reform Commission Grant No. [2014]1507, Shenzhen Peacock Plan
Research Grant KQCX20140521144507925 and Shenzhen
Foundational Research Funding JCYJ20150625142543470.
References

[1] B. Pang, L. Lee, and S. Vaithyanathan, "Thumbs up? Sentiment classification using machine
learning techniques," in Proc. 2002 Conf. Empirical Methods in Natural Language Processing, 2002,
pp. 79-86.

AUGUST 2016 | IEEE Computational intelligence magazine

Table of Contents for the Digital Edition of Computational Intelligence - August 2016