Signal Processing - November 2017 - 73

External knowledge

to determine the set of operations required to answer the
question. A deep neural network is assembled with trained
modules, each corresponding to one of those operations. A
custom network is thus tailored specifically to each question, and finally applied on the image to infer the answer.
A data set of synthetic images named CLEVR (which stands
for compositional language and elementary visual reasoning)
[31] was specifically designed to evaluate generalization to
novel combinations in VQA. It contains photorealistic images
of shapes of various colors and materials. The data set also
contains annotations describing the kind of reasoning that each
question requires (i.e., as functional "programs"). The data set
spurred a series of works on compositional models [29], [32].
The extra an--notations facilitate the training of compositional
models by serving as an intermediate supervision signal. This supervision correspond
VQA models
to an arrangement of modules or operause a monolithic neural
tions to be executed for each question. All
network and end-to-end
of the aforementioned works demonstrated
Modular approaches
supervision to learn
unique capabilities on synthetic data sets.
Most current VQA models use a monolithic
However, it is still unclear how to best apply
neural network and end-to-end supervision
the representations
them to real images and how to train them
to learn the representations of data, the
of data, the reasoning
only using end-to-end supervision, i.e., only
reasoning process, and to capture backprocess, and to capture
knowing the final answer.
ground knowledge from training exambackground knowledge
An alternative approach that addresses
ples. Alternatively, modular approaches
from training examples.
compositionality is the relational networks
have been explored [74], [80] with the
[62]. The idea is to consider the input as a
goal of explicitly factoring the overall proset of objects, such as the locations in a CNN feature map,
cess of VQA into distinct subtasks. The principle of modularand to learn a common predictor that is applied to pairwise
ity allows one to decouple subtasks to some extent, and to
combinations of those objects. The predictor basically learns
use intermediate supervision and leverage several types of
the relations between parts of the input. This proved effective
training data, as opposed to only "end-to-end" question/answer
on the CLEVR data set without the need for the intermediate
pairs. The use of pretrained word embeddings (see the section
supervision mentioned previously.
"Pretraining Language Representations") is a very successful
example of this general principle. Word embeddings are pretrained to capture language-based semantic similarities, and,
Conclusions
in a similar spirit, other representations could be pretrained
This article presented a review of the state of the art on visual
from auxiliary data to capture visual similarities [38] and other
question answering. We reviewed popular approaches based
kinds of background information [71].
on deep learning, which treat the task as a classification probModular systems for VQA also allow decoupling, to some
lem over a set of candidate answers. We described the comdegree, the visual perception from the high-level reasoning.
mon joint embedding model, and additional improvements
For example, Wang et al. [74] proposed a VQA model on top
that build up on this concept, such as attention mechanisms.
of a collection of computer vision algorithms that detect visuDespite shortcomings of current practices for both trainal elements such as objects, persons, and relations between
ing and evaluating VQA systems, we identified a number
them. Thereby, the VQA model only has to reason over this
of promising research avenues that could potentially bring
explicit high-level representation of the contents of the image.
future breakthroughs for both VQA and for the general objective of visual scene understanding.
The setting of the previously mentioned zero-shot VQA
exposes the need for VQA systems to apply to concepts not
present in training question/answers. This motivates the use
of other kinds of data for training, and for retrieving additional information as needed at test time. This requires the
system not only to capture actual information from training
examples, but to learn to retrieve and use novel information,
i.e., learn to learn. That capability of metalearning receives
increased attention [11], [61], [72]. In the context of VQA, [70]
showed the benefit of retrieving on-the-fly, exemplar images
of unknown words from an online search engine. In [75] and
[76], the authors showed the benefit of answering questions
requiring background knowledge of retrieving additional
information from a structured KB. The
extension of these ideas is a promising
Most current
research direction.

Compositional models

The compositional nature of images and language lends
itself to learning similarly compositional models [6]. The
approach aims at addressing the challenge of generalization, i.e., applying the learned model to novel compositions
of words and visual elements. Compositional models were
proposed by Hendricks et al. on the task of image captioning
[27]. Andreas et al. [4], [3], [29] were the first to propose a
compositional architecture for VQA, named neural module
networks. In their approach, the input question is processed

Acknowledgment
All correspondence regarding this article should be addressed
to Qi Wu at qi.wu01@adelaide.edu.au.

Authors
Damien Teney (contact@damienteney.info) obtained his B.
Sc. degree in 2007, his M.Sc. degree in 2009, and his Ph.D.
degree in 2013, all in computer science from the University
of Liege, Belgium. He is a postdoctoral researcher at the

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|

73
http://www.M.Sc
Table of Contents for the Digital Edition of Signal Processing - November 2017
