Signal Processing - November 2017 - 68

What's in the background ?
xiQ

Word
Embeddings

Fixed-Size Vector
Encoders Representations
yQ
BoW, LSTM, ...

W [xi ]
Input Image

Mapping to Joint Space
and Combination
z
Concatenation,
Element-Wise Product,
Bilinear Operator, ...

Scoring of
Candidate Answers

Classifier

CNN

Sky
Mountains
Clouds

Input Question

...

Weighted Sum

Alternative with Attention
Over Image Regions

f att

CNN

... ai Attention Weights
... yi l Region-Specific Features

FIGURE 3. The common approach to VQA is to train a deep neural network for classification over a large set of candidate answers (see the section "Deep
Neural Networks for VQA"). The input question and image are encoded into fixed-size feature vectors (orange bars), using the word embeddings and a
CNN, respectively. The resulting representations are mapped into a joint space, then combined and passed on to the classifier. It assigns scores to a large
set of candidate answers. The top-ranking candidate is returned as the final answer. An attention mechanism (see the section "Attention Mechanisms")
can improve this model and allows the model to focus on relevant parts of the image. In that case, the CNN extracts region-specific image features and
aggregates them using scalar weights (orange squares).

vector encodes the contents of the image. This CNN is typically a standard network architecture that has been pretrained
to perform image recognition [36]. The motivation for a pretrained network is to take advantage of the vast amounts of
training data available for image recognition, relative to the
amounts of data annotated for VQA. The pretrained network is
used as a generic feature extractor, by discarding the final classification layers, and using the features produced within the
CNN prior to this classification [55]. In comparison to classical handcrafted image features such as scale-invariant feature
transform (commonly known as SIFT) [41] or histogram of
oriented gradients (commonly known as HOG) [16], CNN features provide higher-level representations of the contents of the
image, and are naturally produced as a fixed-size vector. The
size of this vector is typically in the order of 1,024 or 2,048.

Question encoding
On the language side, the input question is also processed
to obtain a fixed-size representation of its contents. Initially,
the ith word of the question is represented by an index x Qi in
the input vocabulary. Each word is then turned into a vector. This uses a mapping implemented as a lookup table W [·]
that associates the index of any word of the input vocabulary
to a learned vector. An alternative implementation initially
represents each word with a one-hot vector (a vector of all
zeros, except for a one at the location of the word index in
the vocabulary), which is then multiplied with a dense weight
68

matrix that contains the embeddings of all words. The vecQ
Q
tors of all words W 6x 1 @, W 6x 2 @, f, W 6x QN@ are then collapsed
into a single vector. A simple option for this purpose is to
make a bag-of-words (BoW), which corresponds to simply
averaging the word vectors, i.e., y Q = ^1 N h/ i W 6x Qi @ .
Another popular option is to feed the word vectors into a
recurrent neural network (RNN) such as a long short-term
memory (LSTM). An RNN processes words sequentially
and can capture the sequential relationships between them. In
comparison, a BoW does not account for word order, and, for
example, would produce a same representation for "this man
eats a hot dog" and "a hot man eats this dog."

Combination of image and question features

The feature vectors y I and y Q represent the image and the
questions, respectively. They are each passed through a
learned function before being combined. The intuition here is
to map the features to a joint space, in which distances between
both modalities become comparable. The learned functions
f I($) and f Q ($) are typically implemented as additional layers
of the neural network, e.g., f (y) = Re LU(Wy + b) , where W
and b are learned weights and biases, and ReLU is a rectified
linear unit that serves as a nonlinearity. The mapped features
are then combined before being fed to the output stage. A
simple option for this combination is to simply concatenate
them as z = 6 f I ^ y Ih ; f Q^ y Qh@ . Alternatively, it is popular to
include multiplicative interactions within the neural network

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017