IEEE Signal Processing Magazine - January 2018 - 38

Boat

Person

Dining Table
Object Detection

Semantic Segmentation

Instance Segmentation

Tags: Person, Dining Table

A group of people sitting at
a table

Q: What were the people doing?
A: Eating dinner

Image Classification

Image Captioning

Visual Question-Answering

FIGURE 1. An example of various scene understanding tasks. Some tasks, such as image classification, provide a high-level description of the image by
classifying whether certain tags exist. Other tasks like object detection, semantic segmentation, and instance segmentation provide more detailed and
localized information about the scene. Researchers have also begun to bridge the gap between natural language processing and computer vision with
tasks such as image captioning and visual question-answering.

to localize different objects in an image by placing bounding
boxes around each instance of a predefined object category.
Semantic segmentation, the main focus of this article, aims
for a more precise understanding of the scene by assigning an
object category label to each pixel within the image. Recently,
researchers have also begun tackling new scene understanding
problems such as instance segmentation, which aims to assign
a unique identifier to each segmented object in the image, as
well as bridging the gap between natural language processing
and computer vision with tasks such as image captioning
and visual question-answering, which aim at describing an
image in words, and answering textual questions from images, respectively.
Scene understanding tasks, such as semantic segmentation,
enable computers to extract information from real-world scenarios and leverage this information to accomplish given tasks.
Semantic segmentation has numerous applications including
the following:
■ autonomous vehicles, which need a precise, pixel-level understanding of their environment
■ developing robots, which can navigate and manipulate objects
in their environment
■ diagnosing medical conditions by segmenting cells, tissues, and organs of interest
■ image- and video-editing and developing "smart glasses,"
which describe the scene to the blind.
Semantic segmentation has traditionally been approached
using probabilistic models known as CRFs, which explicitly
model the correlations among the pixels being predicted. However, in recent years, DNNs have been shown to excel at a wide
range of computer vision and machine-learning problems, as
they can automatically learn expressive feature representations
from massive data sets. Despite the representational power
of DNNs, state-of-the-art segmentation algorithms, which are
38

benchmarked on public computer vision data sets and evaluation servers where the test set is withheld, all include a CRF
within their pipelines. Some approaches include CRFs as a
separate stage of the pipeline, while the leading ones incorporate it within the neural network itself.
In this article, we review CRFs and DNNs in the context of
dense, pixelwise prediction tasks and explain how CRFs can
be incorporated into neural networks to combine the advantages of these two models. Markov random fields, CRFs, and,
more generally, probabilistic graphical models are ubiquitous
tools with a long history of applications in a variety of domains
spanning computer vision, computer graphics, and image processing [2]. This is due to their ability to model correlations in
the variables being predicted. DNNs, on the other hand, are
also fast becoming the de facto method of choice in a variety of
machine-learning tasks as they can learn rich feature representations automatically from data. It is, therefore, a natural idea
to combine CRFs with neural networks in a joint framework-
an approach that has been successful in a number of domains.
From a theoretical perspective, it is interesting to investigate
the connections between CRFs and DNNs and explore how
inference algorithms for probabilistic graphical models can be
framed as neural networks themselves. We review CRFs and
DNNs and their integration in the remainder of this article.

CRFs
A naïve way of performing dense prediction tasks like semantic segmentation is to classify each pixel independently using
some features derived from the image. However, such independent pixelwise classification often produces unsatisfactory results that are inconsistent with the visual features in the
image. For example, an independent pixelwise classification
can predict a few spurious incorrect labels in the middle of a
blob of pixels that are classified to have the same label (e.g., a

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
