Signal Processing - November 2017 - 70

The idea in the skip-gram model of word2vec is to
They are interpreted as the importance given by the
train a model which, using the representation (i.e., an embedmodel to each image region.
ding) of a given word, is predictive of the context, i.e., the
The use of an attention mechanism has shown to be very
neighboring words in which it frequently appears [49]. As
beneficial and is now common practice. Variations on this
a consequence, words that are interchangeable or appear
principle have been proposed. For example, [85] and [87]
in similar contexts become associated with similar embeduse multiple rounds of visual attention to allow focusing
dings. Distances between embeddings thus naturally capture
on several regions. In [85], a two-step process performs
semantic relatedness between the words they represent.
a word-guided attention, then a question-guided one. In
More precisely, the skip-gram model seeks to maximize
[65], the authors define image regions with object proposthe ability to predict, from each word embedding, the occurals and then select the regions most to the question and to
rences of other words in a small surrounding window. The
given answer choices. In [43], the authors propose a "hierobjective function to be maximized is
archical coattention" (HieCoAtt) that performs a questionguided attention on the image and an image-guided attention
N
1
on the question.
J= 1 /

/ log p (x j | x i), (1)
N
| X (i) | j ! X (i)
The overall idea of attention in neural networks was initiali
ly motivated by an analogy to the human visual system. Even
though the model is capable of modeling a behavior similar to
where i indexes the N-ordered words in the training corpus,
human attention, this only constitutes an interpretation. In a neux i is the index in the vocabulary of word i , X (i) is a context
ral network-trained end to end, nothing enforces the attention
window of fixed size around word i in the corpus [49]. The
mechanism to actually reflect human-like behavior. In a recent
conditional probability log p (x j | x i) is modeled as a compatstudy [17], Das et al. compared the attention used by human
ibility measure between embeddings such as a dot product
followed by a sigmoid, i.e.,
subjects presented with VQA problems, and VQA m
- odels with
attention [43], [87]. Their conclusion was a
p (x j | x i) = 1/ ^1 + e -W [x i] W [x j] h, (2)

systematically low correlation.
One of the most effective

Pretraining language representations

improvements to the joint
embedding model is to use
visual attention.

As described in the section "Deep Neural Networks for VQA," the first step for
encoding the question is to map words to
vector representations called word embeddings. Each word of
the input vocabulary (i.e., any word appearing in the training
set) is associated with its own embedding, and those embeddings are normally learned alongside the other parameters of
the network via backpropagation. Two potential issues can
arise, however. First, word occurrences in any data set typically follow a long-tailed distribution, meaning that a majority
of words occur infrequently. It is thus difficult to learn stable
and meaningful embeddings for those rare words. Second, the
long-tail property, at its extreme, means that it words commonly appear in test questions that were not seen in any training example. Embeddings for those words cannot be learned
from those examples, and they are typically associated with
an special vector (of zeros or of a special "unknown" token),
and their meaning is practically discarded from the questions.
A solution to these issues is to pretrain word embeddings on a larger auxiliary data set. This practice is known
in the field of natural language processing and has shown
benefit in many tasks besides VQA. Popular methods
for pretraining word embeddings include Global Vectors
for Word Representation [53] (GloVe) and word2vec [48],
which we outline next. The general principle is to use a
large, auxiliary training set of unannotated text, such as
news articles and Wikipedia pages. Those methods require
no specific annotations. That data can thus be much larger
than the training set used for VQA and involve a much
larger vocabulary.
70

where W [·] is a lookup table containing
the embeddings of all words in the vocabulary, reusing the notation of the section
"Deep Neural Networks for VQA." After
the training, the context-prediction part of the model is discarded, and the embeddings associated with the words are
retained (i.e., the table W [·]) and used as word embeddings in
the downstream application such as VQA. The embeddings
can be used as "frozen weights," i.e., static representations
associated with the words, or they can serve as initial values
to be subsequently fine-tuned, i.e., optimized with a lower
learning rate relative to the other network parameters.
Using pretrained embeddings helps the generalization
capabilities of a VQA model. Since semantically similar
words are mapped to close points in the word embedding
space, the processing by the subsequent layers of the network
can more easily 1) interpolate across concepts and 2) generalize to words absent from training questions but for which
embeddings were pretrained.

Memory-augmented neural networks
An active research area is the design of deep neural networks that include an internal memory [13], [52], [66], [77].
Memory-augmented networks have shown success on tasks
such as textual question answering [28], reading comprehension [37], and VQA [83]. The general idea of memoryaugmented networks is to maintain an internal representation
of the input data, on which multiple read and write operations can be applied. The composition of multiple operations
can potentially execute complex chains of inference on the
data. A "controller" part of the network is responsible for

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017