IEEE Signal Processing Magazine - January 2018 - 46

Car

Car
Person
Car

Bus

Input

FCN [6]

DeepLab [5]

CRF-as-RNN [7]

DPN [44]

Higher-Order CRF [33]

FIGURE 7. A comparison of various semantic segmentation methods. FCN tends to produce "blobby" outputs which do not respect the edges of the
image (from the Pascal VOC validation set). DeepLab, which refines outputs of an FCN with DenseCRF, produces an output that is consistent with
edges in the image. CRF-RNN and DPN both train a CRF jointly within a neural network; achieving better results than DeepLab. Unlike other methods,
the higher-order CRF can recover from incorrect segmentation unaries since it also uses cues from an external object detector, while being robust to
false-positive detections (like the incorrect "person" detection). Object detections produced by [46] have been overlaid on the input image, but only
[33] uses this information.

Benefits of end-to-end training
An alternative to unrolling mean-field inference of a CRF and
training the whole network end to end is to train only the "unary"
part of the network and use the CRF as a postprocessing step
whose parameters are determined via cross-validation. We refer
to this approach as disjoint training of the CNN and the CRF,
and show in Table 2 that end-to-end training outperforms disjoint
training in the case of CRF-RNN [7], which only has pairwise
potentials, and the higher-order CRF of [33].
Many recent works in the literature have used CRFs as a postprocessing step. However, as shown in Table 1, the best-performing ones have incorporated the CRF as part of the network itself.
Intuitively, joint training of a CRF with a CNN allows the two
modules to learn to optimally cooperate with each other.

Error analysis
Figures 3 and 7 show that the densely connected pairwise potentials of a CRF improve the segmentation quality at boundaries
of objects. To quantify the improvements that CRFs make to
the overall segmentation, we separately evaluate segmentation performance on the "boundary" and "interior" regions of
the image, as done by [40] and [33]. As shown in Figure 8(c)
and (d), we consider a narrow band (trimap [50]) around the
"void" labels annotated in the VOC 2012 reduced validation
set. The mean IoU of pixels lying within this band is termed

Table 2. A comparison of mean IoU (%) obtained on the VOC 2012
reduced validation set from end to end and disjoint training (adapted
from [33]).

46

Method

Mean IoU [%]

Unary only

68.3

Pairwise CRF trained disjointly

69.5

Pairwise CRF trained end to end

72.9

Higher-order CRF trained disjointly

73.6

Higher-order CRF trained end to end

75.8

the boundary IoU while the interior IoU is evaluated outside
this region.
Figure 8 illustrates our results as the trimap width is varied for FCN [6], CRF-as-RNN [7], DPN [44], and higher-order
CRF [33]. We can see that all of the CRF models improve
the boundary IoU over FCN significantly, although CRF-asRNN and higher-order CRF are almost identical. Moreover,
all of the methods incorporating CRFs also show an improvement in the Interior IoU as well, indicating that the spatial and
appearance consistency encouraged by CRFs is not limited
to only improving segmentation at object boundaries. Furthermore, the higher-order potentials of [33] and [44] show a
substantial increase in interior IoU over CRF-as-RNN, and an
even bigger improvement over FCN. Higher-order potentials
encourage consistency over larger regions of the image [33]
and model contextual relationships between object classes
[44]. As a result, they show larger improvements at the interior
of objects being segmented.

Other examples of unrolling inference algorithms
in neural networks
Unrolling inference algorithms as neural networks is a powerful idea beyond semantic image segmentation. Riegler et al. [51]
presented a method for depth-map superresolution by unrolling the steps of an optimization algorithm and formulating it
as an end-to-end trainable network. Recently, Wang et al. [52]
extended deep structured models to continuous valued output
variables, and addressed tasks such as image denoising and
depth refinement.

Learning arbitrary potentials in CRFs
As the previous section shows, the joint training of a CNN and
a CRF with Gaussian potentials is beneficial for the semantic
segmentation problem. While it is able to output spatially and
appearance-consistent results, some applications may benefit
even more from the usage of more generic potentials. Indeed,
general potentials are able to incorporate more sophisticated
knowledge about a problem than are Gaussian potentials. As an

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
