Signal Processing - November 2017 - 72

accuracy of about 58% to over 70% today (see Tables 1 and 2
for a selection of results). These improvements have been incremental and have now seemed to plateau. In the following, we
examine how current evaluations can mask some inherent issues
of today's approaches and examine promising directions to bring
future breakthroughs.

Issues of data set biases
Several studies have recently pointed out a fundamental issue
with VQA data sets [25], [30], [90]. The text questions alone
often provide strong cues that can be sufficient to answer them
correctly, with no regards to the contents of the input image.
These cues can be obvious. For example, questions starting
with "Do you see a ..." can be correctly answered with a "yes"
almost nine times out of ten [25]. These cue can also stem from
an imbalance among possible answers. For example, questions
starting with "How many ..." often have a correct answer of
"one" or "two" but rarely "17." This issue can also be more
subtle and manifest in the form of conditional biases. For
example, we could imagine that questions starting with "What
is the color ..." can often be answered correctly with "gray"
if it also contains the word "car" and "red" if it contains the
word "flower." Biases conditioned on image contents are also
likely and yet more subtle. Biases are inherent to the real world,
and it is desirable for a VQA model to capture and exploit them
to some extent. However, today's methods have been shown to
overly rely on data set biases and essentially be reduced to rotelearning of training questions. This is counterproductive to the
objective of evaluating visual understanding. A blinded VQA
model (i.e., not being shown the input image, and only guessing
from the question) still achieves an accuracy of 56% versus 65%
in the nonblinded case [30].
The issue of data set biases has been recognized. Attempts
at addressing it include balanced data sets. Zhang et al. [90] first
proposed a data set of clipart images where each binary question is accompanied by two different images that elicit "yes"
and "no" answers, respectively. Goyal et al. applied the idea to
real images, associating two images with each question that lead
to different answers (see example in Figure 2). An appropriate
performance metric in this case is to measure accuracy on pairs

of scenes. Blind models in this case would obtain an accuracy
of 0%, and random guessing 25%. The use of balanced data
sets encourages VQA models, to a larger extent, to utilize visual
information instead of relying on language cues and data set
biases. It is expected that future evaluations of algorithms on
those data sets will be more representative of actual progress on
visual understanding.

Issues with unknown and novel words
A VQA method to be used in a real-world setting, e.g., in
robotics or as personal AI assistants, must be applicable to
open, unrestricted domains. The current paradigm of training
VQA systems with supervision, i.e., with data sets of questions and their ground-truth answers, can only cover a limited
set of objects and concepts. Although VQA data sets have
grown in size, no finite set of exemplars will ever cover the
diversity of objects, actions, relations, etc. in the real world,
for which an ideal VQA system should be prepared. A secondary issue with the current approach is the incentive for
published methods to perform well on benchmark data sets.
These benchmarks do not encourage addressing rare words
and concepts, but rather focus on the concepts most frequent
in the data set. Current methods are therefore designed to
best learn-and often overfit-data set biases.
Recent works have argued for addressing a setting named
zero-shot VQA [54], [70], where questions (or the proposed
multiple-choice answers) specifically involve words that
have not been seen in any training question. For example, a
question "How many zebras are in the image?" may arise,
even though no zebra was involved in the training set. This
setting requires strong generalization capabilities. For example, a related training question "How many giraffes are in
the image?" should be taken as an opportunity to learn to
count, although not giraffes specifically. In parallel of works
on VQA, the learning of high-level reasoning is addressed
in the more abstract setting of program induction (see, e.g.,
[56]). We expect that VQA will ultimately require similar
principled approaches, such as differentiable computing
[26], [50], rather than brute-force learning from limited sets
of examples.

Table 2. A selection of results on the newer VQA v2 data set (test-std split; open-ended questions). Baseline methods score lower on this harder data
set, but the state of the art now reaches more than 70% of accuracy on open-ended questions. The highest accuracies per column are in boldface.
VQA v2 Open Ended

Method

Yes/No

Numbers

Other

All

Baseline: deeper LSTM Q norm. I [42]

73.46

35.18

41.83

54.22

MCB [21]

78.82

38.28

53.36

62.27

UPMC-LIP6 [9]

82.07

41.06

57.12

65.71

Athena [1]

82.50

44.19

59.97

67.59

LV-NUS [1]

81.89

46.29

58.30

66.77

HDU-USYD-UNCC [1]

86.65

51.13

61.75

70.92

Tips and Tricks VQA [2], [68]

86.60

48.64

61.15

70.34

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017