Signal Processing - November 2017 - 47

of supervision in training annotations also improves the segmentation quality in general. Specifically, algorithms using
bounding box annotations show noticeably improved performance compared to the ones based only on an image-level
label. It is because the bounding box provides supervisory
signals for object location and area at the same time, which
are critical to train a model for semantic segmentation, and
totally m
- issing in image-level labels. Also, if used in the right
way, an additional data source like web videos provides useful information for segmentation without additional human
effort since the supervision for segmentation can be reliably
synthesized and effectively transferred from the domain of
the additional data source.
In addition to the weakly supervised approaches, we quantify and compare the results of semisupervised methods. As
shown in Table 2, employing segmentation annotations for
training effectively improves performance even when their
number is very small. It is because the segmentation annotations provide strong and clear guidance to learn semantic
segmentation models reliably from weak labels. Similar to
the weakly supervised cases, the performance of semisupervised approaches depends on the supervision strength of the
employed weak annotations. Also, increasing the amount of
segmentation annotations naturally improves the performance.

Other applications
This survey article has focused on semantic segmentation algorithms for RGB images with various objects. However, weakly
supervised semantic segmentation can be naturally applied to
other tasks as well, and this section introduces a few examples.
Medical image analysis would be an important application of weakly supervised semantic segmentation. Semantic
segmentation of medical images (e.g., segmentation of cancer
lesions) plays an important role in disease recognition and
diagnosis, but collecting segmentation labels for medical
images is particularly expensive because annotators must be
domain experts. Thus, weakly supervised approaches have
attracted much attention even before the era of deep learning [55]. Recently Jia et al. [19] proposed a DCNN-based
approach for histopathology cancer image segmentation.
Similar to [36], a roughly estimated size of a cancerous
region is used as weak supervision in addition to image-level
class labels since it is less costly to obtain relaxed information compared to pixel-level class labels.
Weakly supervised semantic segmentation has been also
applied for autonomous driving. Barnes et al. [3] present a weakly

supervised framework to learn a DCNN segmenting a path proposal and obstacles from a road scene. The path proposal means
the pixel-level area of a route that the vehicle would take, and
manual annotation of such an area is expensive as it is in the typical semantic segmentation setting. In [3], segmentation annotations of those entities are generated with no human intervention
other than driver behavior: a large volume of video data is first
recorded by a camera and a lidar sensor mounted on the vehicle,
then path proposals are annotated by projecting the future path of
the vehicle into each video frame and obstacle areas are derived
from lidar scanning results. Finally, the generated annotations
are used to train a DCNN for segmentation.

Summary and discussion
This article provided a comprehensive review of weakly supervised approaches for semantic segmentation based on DCNNs.
Semantic segmentation aims to estimate pixel-level areas of
semantic categories, and its results enable many interesting
applications that require fine-grained interpretation of an image.
Following their great success in other visual recognition tasks,
DCNNs also have shown impressive performance on public benchmarks for semantic segmentation. However, even with
DCNNs, we still have a long way to go in achieving semantic segmentation in an uncontrolled and realistic environment. Because
of the data-hungry nature of DCNNs and the lack of segmentation annotations for training, fully supervised DCNNs can handle
only a small number of semantic classes that are defined in the
existing training data sets. Weakly supervised approaches tackle
this issue by leveraging readily available or easily obtainable
weak labels instead of segmentation masks. As summarized in
the section "Empirical Analysis," the records achieved by recent
weakly supervised approaches are impressive, especially when
considering the fact that no pixel-level supervision is provided
to train them. However, there is still a certain gap between the
performance of fully supervised approaches and that of weakly
supervised ones.
Here we suggest a few directions worth investigating for
further improvement of weakly supervised semantic segmentation. The first is to get help from unsupervised computervision techniques. Object shape information is essential to
learn semantic segmentation but absent in weak annotations, and we believe that such missing information can be
compensated by the unsupervised techniques. For example,
superpixels and region proposals well preserve underlying
image structures including object boundaries, so they allow
us to bypass explicit shape estimation during inference

Table 2. The comparison results of semisupervised semantic segmentation algorithms on the PASCAL VOC 2012 data set.
Method

Weak Supervision

Number of Segmentation Annotations

mIoU (val)

mIoU (test)

WSSL [35]

Image-level label

0.5K (5% of the training set)

56.9

DecoupledNet [16]

Image-level label

0.5K (5% of the training set)

62.1

62.5

WSSL [35]

Bounding box

1.4K (13% of the training set)

65.1

66.6

Boxsup [9]

Bounding box

1.4K (13% of the training set)

63.5

66.2

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017