Signal Processing - November 2017 - 65

comparison, often requires reasoning with common sense
and with other information not present in the given image. In
this respect, VQA constitutes an AI-complete task [5] since
it requires multimodal knowledge beyond specific domains.
This reinforces the motivation for research on VQA, as it provides a proxy to evaluate progress toward general AI, with
systems capable of advanced reasoning combined with deep
image and language understanding.

VQA-real

The most widely used data set is currently the one proposed
by a team of researchers from Virginia Tech and is commonly
referred to as VQA [5]. It comprises two parts, one using natural
images named VQA-real, and a second one with clipart images named VQA-abstract (discussed at the end of this section).
VQA-real comprises 123,287 training and 81,434 test images,
respectively, sourced from COCO [40]. Human annotators were
encouraged to provide interesting and diverse questions and
short, concise answers (typically two to three words). The data
Data sets for training and evaluating VQA
set allows evaluation both in an open-ended and in a multipleWe now examine data sets that have been specifically compiled
choice setting, the latter providing 17 additional (incorrect) canfor research on VQA. These data sets contain, at a minimum,
didate answers for each question. Overall, the data set contains
triples made each of an image, a question, and its correct answer.
614,163 questions. According to an analysis
Some early data sets were generated semiauperformed by polling annotators, most subtomatically (e.g., from image captions [45])
Despite undeniable
jects (at least six out of ten) estimated that
but modern data sets were created manually
advantages, VQA data sets some common sense was required for 18%
through crowdsourcing [5], [35]. The creation
of clipart images have
of the questions, and adult-level knowledge
of these sets of questions with ground-truth
was necessary for only 5.5% of the quesanswers is very time-consuming, and today's
seen little use compared
largest data sets of several hundreds of thoutions. These figures show that purely visual
to their counterparts of
sands of instances [35] represent a major
information is likely sufficient to answer
real images.
effort. Those data sets are designed for both
most questions.
evaluating and training VQA systems in a
A recent, updated version of this data set,
supervised setting, and the latter demands such large amounts
known as VQA v2.0, includes two images with each question that
of data. As will be discussed in the section "Directions of Current
lead to different answers [25]. This aims at addressing issues of
and Future Research," this very need for large amounts of data is
data set biases.
a significant limit of current approaches.
For the purpose of standardized comparisons and benchVisual genome and visual7W
marking of different algorithms, data sets are split into predeThe Visual Genome QA data set [35] is currently the largest one
termined sets of instances for training, validation, and testing.
designed for VQA, with 1.7 million question/answer pairs. It is
Benchmarks typically do not provide the ground-truth answers
built with images from the Visual Genome project [35], which
of the test set. The evaluation is performed by an automatic
includes structured annotations of scene contents in the form of
online service that compares the provided answers (inferred by
scene graphs. Those scene graphs describe the visual elements
the algorithm to be evaluated) and the private ground truth [5].
of each image with their attributes and the relationships between
This method typically restricts the number and frequency of
them. Human subjects provided questions that must start with
submissions so as to prevent cheating or unintentional overfitone of the seven "Ws"-i.e., who, what, where, when, why,
ting of the test set.
how, and which. The diversity of answers in the Visual Genome
Existing data sets vary mainly along three dimensions 1)
is larger than in VQA-real [5]. The 1,000 most-frequently given
their size, i.e., the number and variety concepts represented
answers in the data set correspond only to the correct answers of
in the images and questions; 2) the amount of required rea64% of all questions. In VQA-real, the corresponding top 1,000
soning, e.g., whether the detection of a single object is sufanswers cover more than 90% of questions. The Visual7w [92]
ficient or whether inference is required over multiple facts
data set is a subset of the Visual Genome that allows evaluation in
or concepts; and 3) how much information beyond what is
a multiple-choice setting, as each question is provided with four
present in the input image is necessary to infer an answer,
plausible but incorrect candidate answers.
e.g., common sense or subject-specific information. Most
data sets lean toward visual-level questions and require little
Zero-shot VQA
external knowledge beyond common sense. These characterA special version of the Visual7W data set was proposed in
istics reflect the fact that current state-of-the-art methods still
[70]. The authors redefined the training and test splits such
struggle with simple visual questions.
that every test instance includes one or several words that
The first VQA data set designed as a benchmark was Data
were not present in any training example. For example, a test
Set for Question Answering on Real World (DAQUAR) for
question "How many zebras are in the image?" might arise
images [45]. The most popular modern data sets [5], [35], [92]
even though the word zebra was never used in the training set.
use images sourced from Microsoft Common Objects in ConThe evaluation of an algorithm with this data set emphasizes
text (COCO), [40] a data set initially devised for image recogits capabilities for generalization beyond training examples
nition, which is itself composed of images from Flickr. Those
and for using sources of information other than VQA-specific
images constitute a very diverse collection of photographs.
data sets. Another similar study appeared in [54].
IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017