IEEE Signal Processing Magazine - January 2018 - 43

the filter. In these works, the last two max-pooling layers were
removed, and Atrous convolutions were used thereafter to ensure
a large receptive field. Note that it is not possible to remove all
max-pooling layers in the network, due to the memory requirements of processing images at full resolution. Other works have
learned more complex networks to upsample the low-resolution
output of an FCN: in [29] an additional "decoder" network is
learned, which progressively "unpools" the initial prediction to
obtain the final full-resolution output. Ghiasi and Fowlkes [30]
learn the basis functions with which to upsample in a coarse-tofine architecture.
Although many architectural innovations have been proposed to improve the segmentation accuracy of neural networks,
they have all benefited from additional refinement by a CRF.
Furthermore, as Table 1 shows, algorithms that have achieved
state-of-the-art results on public benchmarks such as Pascal
VOC [31] have all incorporated CRFs as part of the neural network and trained it jointly with the unary part of the network
end to end [8], [32], [33]. Similar trends are also being observed
on the Cityscapes [34] and ADE20k [35] data sets, which were
released in the last year. Intuitively, the improvement from these
approaches stems from the fact that the parameters of the unary
part of the network, and those of the CRF, may learn to optimally
cooperate with each other.
The rest of this article focuses on these approaches that combine CRFs and CNNs in an end-to-end differentiable network.
We elaborate on how mean-field inference of CRFs can be
unrolled and interpreted as a recurrent neural network (RNN)
in the section "CRFs as RNNs," and in the section "Learning
Arbitrary Potentials in CRFs," we describe other approaches that
enable arbitrary potentials to be learned.

Mean-field inference as a neural network
Chen et al. [5] showed that state-of-the-art semantic segmentation results could be achieved by using the output of an FCN as
the unary potentials of the DenseCRF model of [4]. However,
the CRF was used as postprocessing, and FCN parameters were
learned by backpropagation while CRF parameters were crossvalidated (the authors tried a large number of different CRF
parameters, and finally selected those which gave the highest
performance on a validation set).
This section details how mean-field inference of a DenseCRF
model can be incorporated into the neural network itself, as a
separate "mean-field inference module," an idea that was developed concurrently by Zheng et al. [7] and Schwing and Urtasun [45]. This enables joint training of both the CNN and CRF
parameters by backpropagation. Intuitively, we can expect better
results from this approach as the CNN and CRF learn parameters
which are compatible with each other due to the joint training.
The cross-validation strategy of other works, such as [5], cannot
update the parameters of the CNN such that they are optimal for
the chosen CRF parameters. Zheng et al. named their approach
CRF-as-RNN, and this achieved the best results when that paper
was published.
Mean-field is an iterative algorithm, and crucially for optimization via SGD, the derivative of the output with respect to

Table 1. Results of recent algorithms on the Pascal VOC 2012 test set.
Only the first submission, from 2012, does not use any deep learning.
All of the other methods use a base CNN architecture derived from an
ImageNet pretrained network. Evaluation is performed by a public
server on a withheld test-set. The performance metric is the
Intersection over Union (IoU) [31].
Method

IoU [%]

Base Network

47.8

-

SDS [37]

51.6

AlexNet

FCN [6]

67.2

VGG

Zoom-out [38]

69.6

VGG

DeepLab [5]

71.6

VGG

EdgeNet [39]

73.6

VGG

BoxSup [40]

75.2

VGG

Dilated Conv [27]

75.3

VGG

Centrale Boundaries [41]

75.7

VGG

DeepLab Attention [42]

76.3

VGG

LRR [30]

79.3

ResNet

DeepLab v2 [43]

79.7

ResNet

CRF as RNNs [7]

74.7

VGG

Deep Gaussian CRF [8]

75.5

VGG

Deep parsing network (DPN) [44]

77.5

VGG

Context [32]

77.8

VGG

Higher-order CRF [33]

77.9

VGG

Deep Gaussian CRF [8]

80.2

ResNet

Methods not using deep learning
O2P [36]
Methods not using a CRF

Methods using CRF for postprocessing

Methods with end-to-end CRFs

the input of each iteration can be calculated analytically. Therefore, we can unroll the inference algorithm across its time-steps,
and form an RNN [18]. An RNN is a type of neural network,
usually used to model sequential data, where the output of one
iteration is used as the input of the next iteration and all iterations share the same parameters. In this case, the sequence is
formed from the output of the iterative mean-field inference
algorithm on each time step. When training the network, we
can backpropagate through the RNN and into the previous
CNN to optimize all parameters jointly. Furthermore, as shown
in [7] and described next, for the DenseCRF model, the inference algorithm turns out to consist of standard CNN operations,
making its implementation simple and efficient in standard
neural network libraries. In the section "Incorporating HigherOrder Potentials," we describe how this idea can be extended
beyond DenseCRF to other types of potentials, while the section "Other Examples of Unrolling Inference Algorithms in
Neural Networks" mentions how the idea of unrolling inference
algorithms has subsequently been employed in other domains
using deep learning.

IEEE SIGNAL PROCESSING MAGAZINE

|

January 2018

|

43
Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018
