IEEE Signal Processing Magazine - January 2018 - 29

Pretrained Discriminative CNN

DNN
gφ (.)
y

gφ (y)
ψ (gφ (y))
lfeat = iψ (gφ (y)) - ψ (x)i2

lmse = igφ (y) - xi2
ψ (x)

FIGURE 8. When solving inverse imaging problems, one has the option to use a loss function in the feature space in addition to the loss function in the
original pixel space. In this framework, the features learned by a pretrained discriminative CNN are used to compute the MSE loss in feature space.

with the ground-truth image, but without enforcing pixel-wise
accuracy as dictated by the MSE [42]. The approach of adding a
feature-space loss to the MSE loss was used in SR tasks and was
shown to be particularly successful at reconstructing images with
fine detail and edges (see [24] and [25]).

How do neural networks solve
the inverse problem?
In the section "Neural Networks Architectures for Inverse Problems in Imaging," we reviewed some of the commonly used
deep-learning architectures for solving inverse problems in
imaging. In this section, we provide some insight as to how these
specific choices of models affect the network's approach to solving inverse problems. We show that there exist some tricks, e.g.,
imposing architectural constraints to the model, or incorporating the neural network directly into an analytical method, which
allows a departure from the common notion of a "black-box"
model and guide our model to provide improved solutions to the
imaging problem.

Learning higher-level representations
with encoder-decoder CNNs
In the section "Neural Networks Architectures for Inverse
Problems in Imaging," we have seen two major types of CNNbased architectures used for solving inverse problems in imaging. The first type is composed of multiple convolutional layers
that produce feature maps of fixed size. The other type of CNNs
described consists of an encoding-decoding architecture, in
which case the spatial size of the feature maps first decreases and
then increases again to match the output image size.
For inverse problems for which the input and output images
are of the same size, it may seem unnecessary to first downsam-

ple and then upsample the feature maps. However, there are multiple advantages to using such an encoder-decoder framework for
our inverse problems, as explained by Johnson et al. [42]. First,
decreasing the feature maps at each step of the encoder network
results in fewer algebraic operations performed by the network
which, in turn, increases the efficiency of inference. Second, due
to downsampling, the effective receptive field of the network
increases significantly. The receptive field of a unit in a neural
network refers to the size of the field of view the unit has over its
input layer. For example, the output units in fully connected neural networks have a receptive field that covers the entire input size,
as each output unit is connected to all the units in the input layer.
In CNNs, however, the receptive field of a unit is determined
by the width k of the convolution kernel. The use of successive
convolutions in an encoder-decoder CNN increases the overall
receptive field of the model. The concept of receptive field in
CNNs is of paramount importance in solving inverse problems
such as optical flow or inpainting, since having a large field of
view over the input image can significantly improve the prediction at each pixel in the output image. Therefore, we can use
the large receptive fields provided by encoder-decoder CNNs
to our advantage when solving problems that require such a
wide field of view.
As briefly mentioned in the section "Neural Networks Architectures for Inverse Problems in Imaging," studying the role of
encoder-decoder CNNs from a representation learning perspective can provide us with valuable insight. Encoder-decoder
CNNs can be seen as mapping an input image to a more useful representation, which is then used by the decoder to reconstruct the final image. For example, for the image inpainting
problem, it may be desirable for the neural network to learn
to extract semantic information regarding the input, especially

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018