Signal Processing - November 2017 - 71

c- ontrolling those operations. The mechanism is comparable
to multiple rounds of an attention mechanism, in that it also
enables the modeling of interactions between specific section
of the input data.
The variant proposed in [37] and [83], named dynamic memory networks (DMNs), was successfully applied to VQA. It is built
around four modules (see Figure 5). The input module transforms
the input data into a set of discrete vectors called facts. A question
module computes a vector representation of the question, using
a gated recurrent unit [(GRU), a variant of LSTM]. An episodic
memory module retrieves the facts required to answer the question. A key element is to allow the episodic memory module to
perform multiple passes over the facts to allow transitive reasoning. An attention mechanism selects the relevant facts and an
update mechanism iteratively generates new memory representations from the current state and the retrieved facts. The initial
state is set as the representation produced by the question module.
Finally, the answer module uses the final state of the memory and
the question to predict the final output, using a classic classifier
over candidate answers.

Run time retrieval of additional information
Interfacing a VQA method with external sources of information allows one to separate the reasoning from the representation of prior knowledge in a scalable manner. One
limitation of the basic joint embedding approach is to
attempt to capture all of the information of training examples within the parameters of a neural network. This cannot
scale arbitrarily, however. On one hand, any network has
a finite capacity and, on the other hand, training examples
also provide finite information. Several works explored the
idea of connecting a VQA system with external sources of
information that can be virtually infinite (e.g., web searches) or extensible without needing to retrain the VQA model
(e.g., structured KBs).

In [75] and [82], the authors train a model to interface with a
KB. Such KBs, like DBpedia [7] and Freebase [12], are databases
compiled with facts ranging from common sense to encyclopedic
knowledge. Such nonvisual information can be helpful for VQA.
For example, the question "How many mammals appear in this
image?" requires understanding the word "mammal" and which
animals belong to this category. The VQA system of [75] and
[82] is trained to map the input question/image to queries to be
executed on KBs. The queries retrieve information relevant to the
concepts involved in the question and/or image, which is fed as
an additional input to the output stage of the system. The overall principle has shown limited benefits on existing VQA data
sets, since most questions do not require such specific, nonvisual
information. The idea remains a promising direction for developing scalable VQA systems.
In [70], the authors propose the retrieval of visual information from web searches in the form of exemplar images of question words. Rare and novel words, for example, the name of an
uncommon animal or of an up-and-coming celebrity, are not
likely to appear or be even known during training. The retrieval
of images from the web allows the method to expand its domain
of applicability as needed. The implementation of [70] simply
retrieves the top five images from Google for every word of
the question, from which CNN features are extracted and fed
alongside the input question/image to the VQA system. This
mechanism, however crude, showed an advantage to questions
involving unknown words (i.e., "zero-shot VQA") while leaving
substantial room for future developments; see the section "Issues
with Unknown and Novel Words."

Directions of current and future research
Most modern methods for VQA have been evaluated on the data
set of Antol. et al., which has served as the de facto standard
benchmark. State-of-the-art methods have consistently improved
performance on this data set over the past few years, from an

Final Memory

Episodic Memory Pass 2
Episodic Memory
Attention
Mechanism

Memory
Update

Attention Mechanism

Answer
Palm

AttnGRU

Memory Update

...

AttnGRU

Gate Attention
Question
What kind of
tree is in the
background?

Input Module

Inputs

...

Updated Memory

Episodic Memory Pass 1

AttnGRU

...

Attention Mechanism
...
AttnGRU

Memory Update
AttnGRU

Gate Attention

...

Initial Memory
(a)

(b)

FIGURE 5. DMNs for VQA. (a) The overview and (b) details of the episodic memory module with two passes. (Figure adapted with permission from [83].)
IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017