Signal Processing - November 2017 - 64

questions would likely require information not present in
the image and involve rare words and concepts. In comparison, most questions in -current data sets are purely visual
(e.g., about counts or colors) and centered on common concepts. For example, in one of the most popular data sets [5], a
mere 1,000 different answers can correctly answer more than
90% of questions.
The recent interest in VQA [5], [45], [81] originates from
the latest advances in computer vision on low- and mid-level tasks. This encouraged further research on higher-level
tasks, and the combination of vision with other modalities,
particularly language. Historically, one of the earliest integrations of computer vision with language was the SHRDLU
system dating back to 1972 [78], which allowed the use of
language to instruct a computer to move objects in a simulated "blocks world." Other attempts at creating conversational robotic agents [15], [47], [59] were also grounded in
the visual world. However, these early works were often
limited to specific domains and/or simple language. Deep
learning has now been applied to virtually every problem
imaginable in computer vision, and convolutional neural
networks (CNNs) are approaching human performance in
tasks such as image segmentation [39] or object recognition
[19], [24]. The success of deep learning on perceptual tasks
drove an increasing enthusiasm for high-level tasks. VQA
particularly embodies this confidence in achieving high-level
image understanding.

Task definition and data sets
An instance of VQA consists of an image and a related question given in plain text (see examples in Figure 1). The task
for the machine is to determine the correct answer, which is,
in current data sets, typically a few words or a short phrase.
Two practical variants are usually considered, an open-ended
and a multiple-choice setting [5], [92]. In the latter, a set of
candidate answers are proposed. This makes the evaluation
of a generated answer easier than in the open-ended setting,
where the comparison between the machine's output and a
ground truth (i.e., human provided) answer faces issues with
synonyms and paraphrasing.

Is this pizza vegetarian?

In comparison to classical tasks of computer vision such
as object recognition or image segmentation, instances of
VQA cover a wide range of complexity. Indeed, the question
itself can take an arbitrary form, and so can the set of operations required to answer it. In this sense, VQA more closely
reflects the challenges of general image understanding. VQA
is also related to the task of textual question answering [10],
[14], [88], in which the answer is to be found in a textual narrative (i.e., reading comprehension) or in large knowledge
bases (KBs) (i.e., information retrieval). Textual QA has been
studied for a long time in the natural language processing
(NLP) community, and VQA is basically its extension to a
visual input. The additional challenge of a visual input is significant because images are simply much higher dimensional
than text. Images capture the richness of the real world in a
noisy manner, whereas natural language already represents a
certain level of abstraction. For example, compare the phrase
"a red hat" with the multitude of its representations that one
could picture, e.g., with many different styles and details that
cannot be described in a short phrase.
While, to some extent, the processing of language is
possible with discrete- and rule-based approaches, such as
syntactic parsers and regular expression matching, the complexity of images renders such engineered methods intractable. Modern computer vision is based on statistical learning,
and recent works combining vision and language (including
image captioning and VQA) similarly evolved from machinelearning techniques. Finally, both language and vision are
inherently compositional in their structure. This constitutes
both a challenge and an opportunity when considering the
generalization capabilities of learned models (see the section
"Compositional Models").
Let us mention the relation of VQA with the task of automatic image captioning [20], [73], [79], i.e., generating a
textual description of a given image. It has also attracted
significant interest in the past few years and can be compared to VQA as they both combine vision and language.
The two tasks are complementary as they evaluate different
capabilities. Captioning requires mostly descriptive capabilities that involve almost purely visual information. VQA, in

What is the mustache made of?

Does this person have 20/20 vision?

FIGURE 1. The task of VQA is a significant step toward general AI and a departure from low- and mid-level tasks in classical computer vision. It requires
relating visual concepts with elements of language, common-sense, and general knowledge. (Photos are examples from a major public data set [5].)

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017