Signal Processing - November 2017 - 110

based approaches, which first select a
set of the visually similar images from a
large database and then transfer the capa baby holding a toothbrush
CNN
RNN
tions of retrieved images to fit the query
in its mouth
image [10], [20]. There is little flexibility to modify words based on the content
of the query image, since they directly
FIGURE 1. An illustration of the CNN-RNN-based image captioning framework.
rely on captions of training images and
cannot generate new captions.
Deep neural networks can potentially address both of these
modeling beyond object recognition. In addition, due to the
issues by generating fluent and expressive captions, which can
complexity of scenes in the image, it is difficult to represent all
also generalize beyond those in the train set. In particular, recent
fine-grained, subtle differences among them with the simple
successes of using neural networks in image classification [9],
attribute of category. The supervision for training image caption[15], [23] and object detection [8] have motivated strong interest
ing models is a full description of the content of the image in
in using neural networks for visual captioning.
natural language, which is sometimes ambiguous and lacks finegrained alignments between the subregions in the image and the
words in the description.
Major deep-learning paradigms for image captioning
Moreover, unlike image classification tasks, where we can
easily tell if the classification output is correct or wrong after
The end-to-end framework
comparing it to the ground truth, there are multiple valid ways
to describe the content of an image. It is not easy to tell if the
Vector-to-sequence learning
generated caption is correct or not, at what degree. In practice,
Motivated by the recent success of sequence-to-sequence
human studies are often employed to judge the quality of the
learning in machine translation [37]-[39], researchers studied
caption given an image. However, since human evaluation is
an end-to-end encoder-decoder framework for image captioncostly and time-consuming, many automatic metrics are proing [2]-[4], [12], [26]. Figure 1 illustrates a typical encoderposed, which could serve as a proxy mainly for speeding up the
decoder-based captioning system [26].
development cycle of the system.
In such a framework, first the raw image is encoded by a global
Early approaches to image captioning can be roughly
visual feature vector which represents the overall semantic infordivided into two families. The first one is based on template
mation of the image, via deep CNNs. As illustrated in Figure 2,
matching [6], [16], [17]. These approaches start from detecting
a CNN consists of several convolutional, max-pooling, responseobjects, actions, scenes, and attributes in images and then fill
normalization, and fully connected layers. This architecture has
them into a hand-designed and rigid sentence template. The
been very successful for large-scale image classiﬁcation [21], and
captions generated by these approaches are not always fluent
the learned features have shown to transfer to a broad variety
and expressive. The second family is grounded on retrievalof vision tasks [40]. Usually, given a raw image, the activation
Global Visual
Vector

Caption

5

27
55

256

3

3

3

5

Raw Image

3

3
13
27

384

3
13

13

13

384

13 13
256

1,000
4,096 4,096 Classification
Layer

55
96

Global Visual
Feature Vector

FIGURE 2. An illustration of a deep CNN such as the AlexNet [15]. The CNN is trained for a 1,000-class image classification task on the large-scale ImageNet data set [41]. The last layer of the AlexNet contains 1,000 nodes, each corresponding to a category. The second last fully connected dense layer is
usually extracted as the global visual feature vector, representing the semantic content of the overall images.

110

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|
Table of Contents for the Digital Edition of Signal Processing - November 2017
