Signal Processing - November 2017 - 67

Clipart images
Data sets for VQA have also been proposed with synthetic
clipart images (referred to as abstract scenes in [5]). These
images were created manually with cartoon representations
of characters and objects from a predefined set. The motivation is to enable research on VQA in a controlled setting,
where the computer vision part of the problem is eased by
the restricted set of visual elements. Such data allows focusing on the high-level semantics of the scenes rather than on
visual recognition. For this purpose, the images are provided with structured descriptions, in the form of XML files
that list the objects present in the scene with their visual
properties (e.g., position, scale, etc.). VQA methods can use
these descriptions to completely bypass the visual parsing of
the images.
Using synthetic images gives great control over the elements actually depicted, and this allowed the creation of
a data set of balanced binary questions [90]. That data set
contains only binary (yes/no) questions and each question
appears twice in the data set, with two different images that
give rise to opposite answers. This removes conditional biases that are common in other data sets, for example, a predominance of "yes" answers to questions of the form "Is there ...
in the image?" Those biases otherwise allow to blindly guess
correct answers, which hinders a meaningful evaluation of
VQA systems. Despite undeniable advantages, VQA data sets
of clipart images have seen little use [5], [69], [90] compared
to their counterparts of real images.

Video-based QA
In addition to the studies on image QA mentioned previously,
there have been a few works on VQA with videos. Zhu et al.
[91] assembled a data set of over 100,000 videos and 400,000
questions, using existing collections of videos from different domains, from cooking scenarios to movies and web videos. Tapaswi et al. [67] proposed a setting named MovieQA,
where questions have to be answered using multiple sources
of information including he full-length movies, but also sub-

titles, scripts, and plot summaries. Zeng et al. [89] proposed
the generation of questions from video descriptions.

Evaluation
VQA systems are evaluated by inferring the answers on
the test split of a given data set. Recent data sets [92] recommend the multiple-choice setting, since there is only one
correct answer among the multiple choices. The evaluation
is thus straightforward, as one can simply measure the mean
accuracy over test questions. In an open-ended setting, several answers could be equally valid, because of synonyms
and paraphrasing. This makes a fair evaluation nontrivial.
The usual workaround is to restrict answers, at the time of
the creation of the data sets, to short phrases, typically one
to three words. This restriction limits ambiguities by forcing
questions and answers to be more specific, and allows evaluation by exact string-matching. Most data sets partition the test
questions into subsets depending on the type of answer (e.g.,
yes/no, number, etc.) such that performance can be reported
on each subset (see Table 1).

Deep neural networks for VQA
The common approach to VQA is to train a deep neural network with supervision which maps the given image and question to a relative scoring of candidate answers. The main idea
is to learn a joint embedding of the visual and textual inputs.
First, the image and the question are processed independently to obtain separate vector representations (see Figure 3).
Those features are then are mapped with learned functions to
a joint space, then combined and fed to an output stage. We
examine each of those elements next. The section "Advanced
Techniques" will then look at those techniques that build onto
this model.

Image encoding

On the computer vision side, the input image x I is processed
with a deep convolutional neural network (CNN) to extract
image features described as a vector y I . This large fixed-size

Table 1. A selection of results on the VQA-real data set (test-std split) in both the open-ended and multiple-choice settings.
Performance has incrementally improved over the past few years. The highest accuracies per column are in boldface.
VQA-Real Open Ended
Method

Yes/No

Numbers

Other

Multiple Choice
All

All

Baseline: Deeper LSTM Q norm. I [42]

80.6

36.5

43.7

58.2

63.1

Neural modules networks [4]

81.2

37.7

58.7

Stacked attention networks [87]

58.9

Dynamic memory networks (DMNs+) [83]

60.4

DualNet [60]

81.9

37.8

49.7

61.7

66.7

Hierarchical coattention (HieCoAtt) [43]

62.1

66.1

VQA-machine [74]

81.4

38.2

53.2

63.3

67.8

MLB [34]

37.9

54.8

65.1

68.9

MCB ensemble 7 models [21]

83.2

39.5

66.5

70.1

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017