Signal Processing - November 2017 - 44

weak labels; complicated labels usually provide stronger
formulate semantic segmentation as automatic foreground/
supervision for segmentation while increasing the human
background segmentation within each bounding box area.
annotation cost. To obtain cost-effective labels from a human,
To this end, Papandreou et al. [35] incorporate techniques
some approaches propose utilizing microuser annotation. In
developed for interactive segmentation (i.e., GrabCut [43])
these approaches, a model presents to users multiple candito estimate foreground pixels within the box, where pixels
date masks inferred from an image and asks them to choose
inside and outside the box are considered as initial seeds for
the best mask among the candidates. This process makes the
the foreground and background, respectively. Then, a model
annotation task intuitive and efficient because it needs a simple
for semantic segmentation is trained using the estimated seguser verification by a single click to obtain dense segmentamentation masks as ground truth. On the other hand, instead
tion masks. The success of these approaches is thus heavily
of the direct estimation of pixel-wise labels from the bounddependent on generating diverse and high-quality segmentaing box, Dai et al. [9] exploit off-the-shelf region propostion masks.
als [2]. As the adopted region proposal algorithm provides
Motivated by this, Saleh et al. [44] generate multiple forea candidate set of masks that potentially correspond to an
ground masks by inferring multiple CRF solutions that are
object in a box, the problem is reduced to choosing the best
diverse and have low energy at the same time. Kolesnikov and
region proposal among the ones sufficiently overlapped with
Lampert [21] compute candidate masks
each bounding box. To this end, an iteraby clustering image regions into multiple
tive refinement procedure similar to [35]
The objective of semantic
groups, where each region is described by
is adopted for training, where the model
segmentation is to infer
a feature vector computed by a DCNN.
is trained by pixel-wise annotations comsemantic class labels of
In both of the aforementioned approachputed from the selected region proposals
every pixel in an image.
es, users are asked to select the best mask
and the learned model is, in turn, used to
among the predicted multiple diverse candirefine the proposal selection. Khoreva et
dates, and the selected masks are considered as strong supervial. [20] improve the label prediction within the box by using
the objectness prior [2] as the initial foreground seeds for
sion for learning a semantic segmentation model.
GrabCut segmentation and applying a recursive refinement
procedure as in [9].
Natural language description
Since a bounding box provides incomplete yet sufficiently
A natural language description of an image can be used as an
strong supervision for object location and area missing in imageannotation since it provides comprehensive information of the
level class labels, all of the approaches based on bounding box
image including object attributes, relations between objects,
annotations substantially improve the performance over the
scene context, and so on. Also, such a description is readily
ones trained only with the image-level labels. Moreover, they
available for a large number of images found on photo-sharing
are even competitive to the fully supervised counterpart.
sites like Flickr.
Lin et al. [28] exploits natural language image description
as weak annotation. Specifically, they propose performing
Scribble
semantic segmentation by aligning the semantic structures
A scribble is a line in an arbitrary form obtained by a single
of image description and image regions. To this end, both
user stroke, and as another form of weak annotation, it provides
an image and its description are parsed into tree structures
sparse information about object location and extent. One can
through independent procedures; the tree for the description
consider a scribble as a middle ground of point- and box-level
follows the grammatical structure of sentences, and that of the
annotations since a point is a special case of a scribble (i.e.,
image, obtained by a recurrent neural network, defines a hiera scribble with zero length) and a scribble roughly indicates
archical structure of image segments discovered by a semanobject area as a bounding box does. Scribbles provide not only
tic segmentation model. The segmentation model is trained to
a user-friendly way for annotation but also an easier way to
align the two parsing trees.
localize objects in arbitrary shapes. Since the scribble covers
only a partial area of a semantic entity, the inference of pixelwise labels is reduced to propagating the annotated labels to
Additional data source
unmarked pixels. Lin et al. [26] formulate the label propagation
Some of the aforementioned annotation types-point, scribble,
as an optimization problem based on a graphical model, where
and microuser annotation-are not readily available in existing
vertices of the graph are superpixels of each image. Training is
large-scale data sets and demand a certain level of human interperformed by alternating label estimation and model parameter
vention. Although such types of annotations are much easier to
learning, where the model is trained under the supervision of
obtain than pixel-wise labels, their demands for human intervensuperpixel labels and, in turn, used to update the labels as a part
tion is not desirable when considering that the main motivation
of the graph-based optimization procedure.
of weakly supervised learning is to reduce human intervention
required for training. To incorporate stronger supervision without extra human labeling effort, some approaches propose the
Microuser annotation
exploitation of an additional source of data, which are freely
As described previously, there exists a tradeoff between annoavailable in other data sets or different data domains.
tation cost and the amount of supervision in the selection of
44

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017