IEEE Computational Intelligence Magazine - August 2018 - 58

potato" or named entities such as "Boston Globe") does not represent the
combination of meanings of individual
words. One solution to this problem, as
explored by Mikolov et al. [3], is to
identify such phrases based on word
co-occurrence and train embeddings for
them separately. More recent methods
have explored directly learning n-gram
embeddings from unlabeled data [22].
Another limitation comes from
learning embeddings based only on a
small window of surrounding words,
sometimes words such as good and bad
share almost the same embedding [23],
which is problematic if used in tasks
such as sentiment analysis [24]. At times
these embeddings cluster semanticallysimilar words which have opposing sentiment polarities. This leads the downstream model used for the sentiment
analysis task to be unable to identify this
contrasting polarities leading to poor
performance. Tang et al. [25] addresses
this problem by proposing sentiment
specific word embedding (SSWE).
Authors incorporate the supervised sentiment polarity of text in their loss functions while learning the embeddings.
A general caveat for word embeddings
is that they are highly dependent on the
applications in which it is used. Labutov
and Lipson [26] proposed task specific
embeddings which retrain the word
embeddings to align them in the current
task space. This is very important as training embeddings from scratch requires large
amount of time and resource. Mikolov et
al. [8] tried to address this issue by proposing negative sampling which is frequencybased sampling of negative terms while
training the word2vec model.
Traditional word embedding algorithms assign a distinct vector to each

word. This makes them unable to
account for polysemy. In a recent work,
Upadhyay et al. [27] provided an innovative way to address this deficit. The
authors leveraged multilingual parallel
data to learn multi-sense word embeddings. For example, the English word
bank, when translated to French provides
two different words: banc and banque
representing financial and geographical
meanings, respectively. Such multilingual
distributional information helped them
in accounting for polysemy.
Table 1 provides a directory of existing
frameworks that are frequently used for
creating embeddings which are further
incorporated into deep learning models.
C. Character Embeddings

Word embeddings are able to capture
syntactic and semantic information, yet
for tasks such as POS-tagging and NER,
intra-word morphological and shape
information can also be very useful. Generally speaking, building natural language
understanding systems at the character
level has attracted certain research attention [28]-[31]. Better results on morphologically rich languages are reported in
certain NLP tasks. Santos and Guimaraes
[30] applied character-level representations, along with word embeddings for
NER, achieving state-of-the-art results in
Portuguese and Spanish corpora. Kim et
al. [28] showed positive results on building a neural language model using only
character embeddings. Ma et al. [32]
exploited several embeddings, including
character trigrams, to incorporate prototypical and hierarchical information for
learning pre-trained label embeddings in
the context of NER.
A common phenomenon for languages with large vocabularies is the

TABLe 1 Frameworks providing embedding tools and methods.
FrAmework

LAnguAge

urL

S-SPACE

JAvA

HTTPS://giTHuB.CoM/fozziETHEBEAT/S-SPACE

SEMANTiCvECToRS

JAvA

HTTPS://giTHuB.CoM/SEMANTiCvECToRS/

gENSiM

PYTHoN

HTTPS://RADiMREHuREk.CoM/gENSiM/

PYDSM

PYTHoN

HTTPS://giTHuB.CoM/JiMMYCALLiN/PYDSM

DiSSECT

PYTHoN

HTTP://CLiC.CiMEC.uNiTN.iT/CoMPoSES/TooLkiT/

fASTTExT

PYTHoN

HTTPS://fASTTExT.CC/

ieee Computational intelligenCe magazine | august 2018

unknown word issue or out-of-vocabular y word (OOV) issue. Character
embeddings naturally deal with it since
each word is considered as no more than
a composition of individual letters. In
languages where text is not composed of
separated words but individual characters
and the semantic meaning of words map
to its compositional characters (such as
Chinese), building systems at the character level is a natural choice to avoid word
segmentation [33]. Thus, works employing deep learning applications on such
languages tend to prefer character
embeddings over word vectors [34]. For
example, Peng et al. [35] proved that radical-level processing could greatly
improve sentiment classification performance. In particular, the authors proposed two types of Chinese radical-based
hierarchical embeddings, which incorporate not only semantics at radical and
character level, but also sentiment information. Bojanowski et al. [36] also tried
to improve the representation of words
by using character-level information in
morphologically-rich languages. They
approached the skip-gram method by
representing words as bag-of-characters
n-grams. Their work thus had the effectiveness of the skip-gram model along
with addressing some persistent issues of
word embeddings. The method was also
fast, which allowed training models on
large corpora quickly. Popularly known
as FastText, such a method stands out
over previous methods in terms of speed,
scalability, and effectiveness.
Apart from character embeddings,
other approaches have been proposed for
OOV handling. Herbelot and Baroni [37]
provided OOV handling on-the-fly by
initializing the unknown words as the
sum of the context words and refining
these words with a high learning rate.
However, their approach is yet to be
tested on typical NLP tasks. Pinter et al.
[38] provided an interesting approach of
training a character-based model to recreate pre-trained embeddings. This allowed
them to learn a compositional mapping
from character to word embedding, thus
tackling the OOV problem.
Despite the ever growing popularity
of distributional vectors, recent discussions

HTTPS://giTHuB.CoM/fozziETHEBEAT/S-SPACE HTTPS://giTHuB.CoM/SEMANTiCvECToRS/ HTTPS://RADiMREHuREk.CoM/gENSiM/ HTTPS://giTHuB.CoM/JiMMYCALLiN/PYDSM http://clic.cimec.unitn.it/composes/toolkit/ HTTPS://fASTTExT.CC/

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2018