IEEE Signal Processing Magazine - January 2018 - 42

modifications, be adapted for other scene understanding tasks
such as object detection and semantic segmentation. Furthermore, it was possible for Girschick et al. to fine-tune their network from a network already trained on ImageNet since most
of the layers were the same. Fine-tuning from an existing ImageNet-pretrained model provided better parameter-initialization
for training via backpropagation, and has been found to improve
performance in many computer vision tasks. Therefore, the work
of Girshick et al. suggested that CNNs for semantic segmentation
should be based on ImageNet trained architectures as well.
A key idea to extending CNNs designed for image classification to other more complex tasks such as semantic segmentation
is realizing that a fully connected layer can be considered as a
convolutional layer, where the filter size is the same as the size of
the input feature map [6], [26]. Long et al. [6] converted the fully
connected layers of common architectures such as AlexNet [21]
and VGG [22] into convolutional layers, and named them fully
convolutional networks (FCNs). Since these networks consist
of only convolutional-, pooling- and ReLU nonlinearity layers,
they can operate on any arbitrarily sized image. However due to
max-pooling in the network, the output would be a downsampled
version of the input, as shown in Figure 4. Common architectures
such as AlexNet [21], VGG [22] and ResNet [23] all consist of
five pooling layers of size 2 # 2, and hence, the output is downsampled by a factor of 32 in these FCNs. Long et al. showed that
even by simply bilinearly upsampling the coarse predictions up
to the original size of the image, state-of-the-art performance at
the time of publication could be achieved. This method is simple

to implement, can be initialized with the parameters of a CNN
trained on ImageNet, and then be fine-tuned on smaller data sets,
which significantly improves results over initializing with random weights.
Although the fully convolutional approach of Long et al.
achieved state-of-the-art performance, the predictions of the
model were still quite coarse and "blobby," since the max-pooling stages in earlier parts of the network resulted in a lot of spatial information being lost. As a result, fine structures and object
boundaries were usually segmented poorly. This has led to a lot
of follow-up work on improving the segmentation performance
of neural networks.
Chen et al. [5] used the outputs of a CNN as the unary potentials of a DenseCRF model, and showed that applying a CRF
as postprocessing on these unaries could significantly improve
results and provide sharper boundaries [as shown in Figure 3(c)].
In fact, the absolute performance improvement from applying
DenseCRF on CNN unaries was greater than that of Textonboost
[3] unaries [5]. Other works have improved the CNN architecture
by addressing the loss of resolution caused by max-pooling. It
is not possible to completely remove max-pooling from a CNN
architecture for segmentation, since it will mean that layers deeper down will not have sufficient context or receptive field to make
a good prediction. To combat this issue, Atrous [5] or Dilated [27]
convolutions have been proposed (inspired by the "algorithme à
trous" used in computing the undecimated wavelet transform
[28]), which enables the receptive field of a convolution filter
to be increased without increasing the number of parameters in

00
1,

6
38
4

"Tabby Cat"

Convolutionalization

25
6

38
4
38
4
25
6
4,
09
4, 6
09
6
1,
00
0

Tabby Cat Heat Map

FIGURE 4. FCNs. Fully connected layers can easily be converted into convolutional layers by recognizing that a fully connected layer is simply a convolutional layer where the size of the convolutional filter and input feature map are identical. This enables CNNs trained for image classification to output
a coarse segmentation when input a larger image. This simple method enables good initialization and efficient training of CNNs for pixelwise prediction.
(Figure used with permission from [6].)

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018