Signal Processing - November 2017 - 41

Full Supervision

Weak Supervision
Train
Train

Train
Train
Person

Person
Person

Bird

Bird

Person

Bird
Bird

Image

Pixel-Level
Labels

Image-Level
Class Labels

Points

Bounding
Boxes

Scribbles

FIGURE 2. Illustrations of various weak annotations employed for weakly supervised semantic segmentation.

i-ndicating object location and shape is critical in learning to
predict segmentation masks but is partly or totally absent in
weak supervision. The success of weakly supervised approaches depends heavily on the way to compensate the missing information during training.
The purpose of this article is to provide an introduction
to weakly supervised semantic segmentation and a thorough
review of recent approaches in this line of research. In particular, we narrow our focus to approaches based on DCNNs.
There exist weakly supervised approaches proposed before
the era of deep learning [50]-[52], [56]. They attempt to associate pixels with image-level class labels by first computing
region-based classification scores and further refining them
using similarities between local image regions based on various handcrafted visual cues. However, their performance is, in
general, limited due to lack of robust appearance models. On
the other hand, DCNNs provide more natural ways to associate pixels with image labels and more robust feature representations for appearance modeling. In addition, DCNNs are
flexible enough to integrate various types of weak supervision
and additional information that may be useful to improve segmentation performance.

DCNNs for semantic segmentation
This section provides an overview of approaches for semantic segmentation based on DCNNs. The objective of semantic
segmentation is to infer semantic class labels of every pixel
in an image. To achieve this goal, many existing approaches
pose the task as dense local area classification and modify a
DCNN designed for image classification to predict class scores
for every local area in an input image.
The most popular choice of network architecture in this
direction is a fully convolutional network (FCN) [32]. The
FCN is based on a DCNN pretrained for large-scale image
classification, but its architecture is fully convolutional (i.e.,

(a)

(b)

FIGURE 3. Illustrations of popular DCNN architectures used for semantic segmentation. (a) A fully convolutional network makes all network
components fully convolutional, thus converts a CNN trained for image
classification to produce class scores on local image regions. Optional
skip-connections from lower layers (dashed lines) are used to reconstruct
spatial information lost by spatial poolings. (b) The network architecture
of a deep convolutional encoder-decoder network. On top of the convolutional network, a stack of deconvolutional layers are used to reconstruct
fine object segmentation masks using many network parameters.

having no fully connected layer) as it interprets fully connected
layers of the classification DCNN as 1 × 1 convolution filters
so that it can handle input images with arbitrary sizes. The output of the network then has a form of a class score map over
the image. Since the output score map is low resolution, due to
multiple pooling operations in the network, a single deconvolution layer is employed on top of the class score map to enlarge
the size of the output map to that of the input image. The overall network architecture of the FCN is illustrated in Figure 3(a).
Since the output of the network corresponds to pixel-wise class
prediction scores, it is possible to learn the entire model parameters in an end-to-end manner by computing classification loss
on every pixel location using pixel-level ground-truth labels.

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|

41
Table of Contents for the Digital Edition of Signal Processing - November 2017
