IEEE Signal Processing Magazine - January 2018 - 48

CNN

Ground Truth
Right
Chest

CNN
+
General CRF

Right
Upper
Arm

15
10
5
0
-5
-10

0.12
0.08
0.04
0.00
-0.04
-0.08
-0.12

y-Shift

Input Depth

CNN
+
Dense CRF

Left
Torso

-15
-10
-5
0
5
10
15

-15

x-Shift
(b)

Right
Torso

15
10
5
0
-5
-10

0.06

y-Shift

0.04
0.02
0.00
-0.02
-0.04

-15
-10
-5
0
5
10
15

-15

x-Shift
(c)

(a)

FIGURE 9. Human body parts segmentation with generic pairwise potentials. (a) From left to right: the input depth image; the corresponding ground-truth labeling
for all body parts; the result of a trained CNN model; the result of CRF-as-RNN [7], where the pairwise potentials are a sum of Gaussian kernels; and the result of
[53] that jointly train CNN and CRF with general parametric potentials. Note how the hands and elbows are segmented better. (b) Weights for pairwise potentials
that connect the label "left torso" with the label "right torso." Red means a high energy value, i.e., a discouraged configuration, while blue means the opposite. The
potentials enforce a straight, vertical border between the two labels, i.e., there is a large penalty for "left torso" on top (or below) of "right torso" (x-shift 0, y-shift
arbitrary). Also, it is encouraged that "right torso" is to the right of the "left torso" (Positive x-shift and y-shift 0). (c) Weights for pairwise potentials that connect the
label "right chest" with the label "right upper arm." It is discouraged that the "right upper arm" appears close to "right chest," but this configuration can occur at a
certain distance. Since the training images have no preferred arm-chest configurations, all directions have similar weights. Such relations between different parts
cannot by the Gaussian kernels used in CRF-as-RNN.

2) the number of edges in each graphical model, and 3) the number of possible variable configurations, which can be assigned to
each graph node. In total, this typically requires one or even two
orders of magnitude more storage than the training set itself. So
far, this method has not been shown on large training sets.

Learning by backpropagating through inference
Formulating the steps of CRF inference as differentiable operations enable both the learning and inference of the CRF to be
incorporated in a standard deep-learning framework, without
having to explicitly compute marginals. An example of this was
discussed in the section "CRFs as RNNs."
However, in contrast to that approach, we can look at the inference problem from a discrete optimization point of view. Here, it
can be seen as an integer program where a given cost should be
minimized while satisfying a set of constraints (each pixel should
be assigned one label). An alternative inference approach is to
do a continuous relaxation (allowing variables to be real-valued
instead of integers) of this integer program and search for a feasible minimum. The solution to the original discrete problem can
then be derived from the solution to the relaxed problem. Desmaison et al. [55] presented methods to solve several continuous
relaxations of this problem. They showed that these methods generally achieve lower energies than mean-field-based approaches.
48

In the work of Larsson et al. [54], a projected gradient method
based on only differential operations is presented. This inference
method minimizes a continuous relaxation of the CRF energy
and the weights can be learned by backpropagating the error
derivative through the gradient steps. An example of the pairwise
potentials learned by this method is shown in Figure 10. Chandra
and Kokkinos [8] solve the energy minimization problem by formulating it as a convex quadratic program and finding the global
minimum by solving a linear system. As seen in Table 1, it is the
best-published approach on the PASCAL VOC 2012 data set at
the time of this writing.

Methods based on discriminative learning
There is another discriminative learning technique that does
not require the computation of marginal distributions. This
class of methods formulate the learning as a structured support
vector machine (SSVM), which is an extension to the support
vector machine allowing structured output. Since solving these
usually requires doing inference for several setups of weights
an efficient inference method is crucial. Larsson et al. [56] utilized this for medical image segmentation doing inference with
a highly efficient graph-cut method. Knöbelreiter et al. [57] proposes a very efficient GPU-parallelized energy minimization
solver to efficiently perform inference for the stereo problem.

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018