IEEE Computational Intelligence Magazine - August 2018 - 58
potato" or named entities such as "Boston Globe") does not represent the
combination of meanings of individual
words. One solution to this problem, as
explored by Mikolov et al. [3], is to
identify such phrases based on word
co-occurrence and train embeddings for
them separately. More recent methods
have explored directly learning n-gram
embeddings from unlabeled data [22].
Another limitation comes from
learning embeddings based only on a
small window of surrounding words,
sometimes words such as good and bad
share almost the same embedding [23],
which is problematic if used in tasks
such as sentiment analysis [24]. At times
these embeddings cluster semanticallysimilar words which have opposing sentiment polarities. This leads the downstream model used for the sentiment
analysis task to be unable to identify this
contrasting polarities leading to poor
performance. Tang et al. [25] addresses
this problem by proposing sentiment
specific word embedding (SSWE).
Authors incorporate the supervised sentiment polarity of text in their loss functions while learning the embeddings.
A general caveat for word embeddings
is that they are highly dependent on the
applications in which it is used. Labutov
and Lipson [26] proposed task specific
embeddings which retrain the word
embeddings to align them in the current
task space. This is very important as training embeddings from scratch requires large
amount of time and resource. Mikolov et
al. [8] tried to address this issue by proposing negative sampling which is frequencybased sampling of negative terms while
training the word2vec model.
Traditional word embedding algorithms assign a distinct vector to each
word. This makes them unable to
account for polysemy. In a recent work,
Upadhyay et al. [27] provided an innovative way to address this deficit. The
authors leveraged multilingual parallel
data to learn multi-sense word embeddings. For example, the English word
bank, when translated to French provides
two different words: banc and banque
representing financial and geographical
meanings, respectively. Such multilingual
distributional information helped them
in accounting for polysemy.
Table 1 provides a directory of existing
frameworks that are frequently used for
creating embeddings which are further
incorporated into deep learning models.
C. Character Embeddings
Word embeddings are able to capture
syntactic and semantic information, yet
for tasks such as POS-tagging and NER,
intra-word morphological and shape
information can also be very useful. Generally speaking, building natural language
understanding systems at the character
level has attracted certain research attention [28]-[31]. Better results on morphologically rich languages are reported in
certain NLP tasks. Santos and Guimaraes
[30] applied character-level representations, along with word embeddings for
NER, achieving state-of-the-art results in
Portuguese and Spanish corpora. Kim et
al. [28] showed positive results on building a neural language model using only
character embeddings. Ma et al. [32]
exploited several embeddings, including
character trigrams, to incorporate prototypical and hierarchical information for
learning pre-trained label embeddings in
the context of NER.
A common phenomenon for languages with large vocabularies is the
TABLe 1 Frameworks providing embedding tools and methods.
FrAmework
LAnguAge
urL
S-SPACE
JAvA
HTTPS://giTHuB.CoM/fozziETHEBEAT/S-SPACE
SEMANTiCvECToRS
JAvA
HTTPS://giTHuB.CoM/SEMANTiCvECToRS/
gENSiM
PYTHoN
HTTPS://RADiMREHuREk.CoM/gENSiM/
PYDSM
PYTHoN
HTTPS://giTHuB.CoM/JiMMYCALLiN/PYDSM
DiSSECT
PYTHoN
HTTP://CLiC.CiMEC.uNiTN.iT/CoMPoSES/TooLkiT/
fASTTExT
PYTHoN
HTTPS://fASTTExT.CC/
58
ieee Computational intelligenCe magazine | august 2018
unknown word issue or out-of-vocabular y word (OOV) issue. Character
embeddings naturally deal with it since
each word is considered as no more than
a composition of individual letters. In
languages where text is not composed of
separated words but individual characters
and the semantic meaning of words map
to its compositional characters (such as
Chinese), building systems at the character level is a natural choice to avoid word
segmentation [33]. Thus, works employing deep learning applications on such
languages tend to prefer character
embeddings over word vectors [34]. For
example, Peng et al. [35] proved that radical-level processing could greatly
improve sentiment classification performance. In particular, the authors proposed two types of Chinese radical-based
hierarchical embeddings, which incorporate not only semantics at radical and
character level, but also sentiment information. Bojanowski et al. [36] also tried
to improve the representation of words
by using character-level information in
morphologically-rich languages. They
approached the skip-gram method by
representing words as bag-of-characters
n-grams. Their work thus had the effectiveness of the skip-gram model along
with addressing some persistent issues of
word embeddings. The method was also
fast, which allowed training models on
large corpora quickly. Popularly known
as FastText, such a method stands out
over previous methods in terms of speed,
scalability, and effectiveness.
Apart from character embeddings,
other approaches have been proposed for
OOV handling. Herbelot and Baroni [37]
provided OOV handling on-the-fly by
initializing the unknown words as the
sum of the context words and refining
these words with a high learning rate.
However, their approach is yet to be
tested on typical NLP tasks. Pinter et al.
[38] provided an interesting approach of
training a character-based model to recreate pre-trained embeddings. This allowed
them to learn a compositional mapping
from character to word embedding, thus
tackling the OOV problem.
Despite the ever growing popularity
of distributional vectors, recent discussions
HTTPS://giTHuB.CoM/fozziETHEBEAT/S-SPACE
HTTPS://giTHuB.CoM/SEMANTiCvECToRS/
HTTPS://RADiMREHuREk.CoM/gENSiM/
HTTPS://giTHuB.CoM/JiMMYCALLiN/PYDSM
http://clic.cimec.unitn.it/composes/toolkit/
HTTPS://fASTTExT.CC/
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2018
Contents
IEEE Computational Intelligence Magazine - August 2018 - Cover1
IEEE Computational Intelligence Magazine - August 2018 - Cover2
IEEE Computational Intelligence Magazine - August 2018 - Contents
IEEE Computational Intelligence Magazine - August 2018 - 2
IEEE Computational Intelligence Magazine - August 2018 - 3
IEEE Computational Intelligence Magazine - August 2018 - 4
IEEE Computational Intelligence Magazine - August 2018 - 5
IEEE Computational Intelligence Magazine - August 2018 - 6
IEEE Computational Intelligence Magazine - August 2018 - 7
IEEE Computational Intelligence Magazine - August 2018 - 8
IEEE Computational Intelligence Magazine - August 2018 - 9
IEEE Computational Intelligence Magazine - August 2018 - 10
IEEE Computational Intelligence Magazine - August 2018 - 11
IEEE Computational Intelligence Magazine - August 2018 - 12
IEEE Computational Intelligence Magazine - August 2018 - 13
IEEE Computational Intelligence Magazine - August 2018 - 14
IEEE Computational Intelligence Magazine - August 2018 - 15
IEEE Computational Intelligence Magazine - August 2018 - 16
IEEE Computational Intelligence Magazine - August 2018 - 17
IEEE Computational Intelligence Magazine - August 2018 - 18
IEEE Computational Intelligence Magazine - August 2018 - 19
IEEE Computational Intelligence Magazine - August 2018 - 20
IEEE Computational Intelligence Magazine - August 2018 - 21
IEEE Computational Intelligence Magazine - August 2018 - 22
IEEE Computational Intelligence Magazine - August 2018 - 23
IEEE Computational Intelligence Magazine - August 2018 - 24
IEEE Computational Intelligence Magazine - August 2018 - 25
IEEE Computational Intelligence Magazine - August 2018 - 26
IEEE Computational Intelligence Magazine - August 2018 - 27
IEEE Computational Intelligence Magazine - August 2018 - 28
IEEE Computational Intelligence Magazine - August 2018 - 29
IEEE Computational Intelligence Magazine - August 2018 - 30
IEEE Computational Intelligence Magazine - August 2018 - 31
IEEE Computational Intelligence Magazine - August 2018 - 32
IEEE Computational Intelligence Magazine - August 2018 - 33
IEEE Computational Intelligence Magazine - August 2018 - 34
IEEE Computational Intelligence Magazine - August 2018 - 35
IEEE Computational Intelligence Magazine - August 2018 - 36
IEEE Computational Intelligence Magazine - August 2018 - 37
IEEE Computational Intelligence Magazine - August 2018 - 38
IEEE Computational Intelligence Magazine - August 2018 - 39
IEEE Computational Intelligence Magazine - August 2018 - 40
IEEE Computational Intelligence Magazine - August 2018 - 41
IEEE Computational Intelligence Magazine - August 2018 - 42
IEEE Computational Intelligence Magazine - August 2018 - 43
IEEE Computational Intelligence Magazine - August 2018 - 44
IEEE Computational Intelligence Magazine - August 2018 - 45
IEEE Computational Intelligence Magazine - August 2018 - 46
IEEE Computational Intelligence Magazine - August 2018 - 47
IEEE Computational Intelligence Magazine - August 2018 - 48
IEEE Computational Intelligence Magazine - August 2018 - 49
IEEE Computational Intelligence Magazine - August 2018 - 50
IEEE Computational Intelligence Magazine - August 2018 - 51
IEEE Computational Intelligence Magazine - August 2018 - 52
IEEE Computational Intelligence Magazine - August 2018 - 53
IEEE Computational Intelligence Magazine - August 2018 - 54
IEEE Computational Intelligence Magazine - August 2018 - 55
IEEE Computational Intelligence Magazine - August 2018 - 56
IEEE Computational Intelligence Magazine - August 2018 - 57
IEEE Computational Intelligence Magazine - August 2018 - 58
IEEE Computational Intelligence Magazine - August 2018 - 59
IEEE Computational Intelligence Magazine - August 2018 - 60
IEEE Computational Intelligence Magazine - August 2018 - 61
IEEE Computational Intelligence Magazine - August 2018 - 62
IEEE Computational Intelligence Magazine - August 2018 - 63
IEEE Computational Intelligence Magazine - August 2018 - 64
IEEE Computational Intelligence Magazine - August 2018 - 65
IEEE Computational Intelligence Magazine - August 2018 - 66
IEEE Computational Intelligence Magazine - August 2018 - 67
IEEE Computational Intelligence Magazine - August 2018 - 68
IEEE Computational Intelligence Magazine - August 2018 - 69
IEEE Computational Intelligence Magazine - August 2018 - 70
IEEE Computational Intelligence Magazine - August 2018 - 71
IEEE Computational Intelligence Magazine - August 2018 - 72
IEEE Computational Intelligence Magazine - August 2018 - 73
IEEE Computational Intelligence Magazine - August 2018 - 74
IEEE Computational Intelligence Magazine - August 2018 - 75
IEEE Computational Intelligence Magazine - August 2018 - 76
IEEE Computational Intelligence Magazine - August 2018 - Cover3
IEEE Computational Intelligence Magazine - August 2018 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com