IEEE Signal Processing Magazine - January 2018 - 31

ways to incorporate domain expertise within the neural network framework. We describe how, with carefully designed
choices, we can guide our neural networks to learn operations
similar to those implemented by analytical models. This may
be seen as a way to incorporate domain expertise within the
DNN framework. Similarly to the residual learning approach
described in the section "Facilitating the Learning Process:
Gradual Refinement with Deep Residual CNNs," guiding the
neural network to perform known analytical steps may facilitate the learning procedure of the models, hence resulting in
better solutions.
Nonblind deconvolution, e.g., is an inverse problem for
which we are provided with domain knowledge (more specifically here the knowledge of the degradation system operating
on the original image) that should not be ignored when designing the inverse system. Xu et al. [45] propose to inject their
knowledge of the blur kernel directly into the weights of their
neural network architecture. More specifically, they use the
singular value decomposition of the pseudo-inverse of their
blur kernel to initialize the weights of their large 1-D convolution kernels. This provides their model with a good starting
point to learn an operation similar to nonblind inverse filtering. Wang et al. [46] choose to design a multilayer neural network that mimics the operations implemented by the unfolded
LISTA for SR. The authors use their knowledge of each step
implemented by the LISTA algorithm [47] to fix their initial
weights to precomputed values. Both Xu et al. [45] and Wang
et al. [46] show that making an explicit choice of weight initialization using knowledge of another algorithm improves
the final performance of their model compared with random
weight initialization.
LISTA [47] represents a simple case of an unfolding algorithm [48] that aims to combine the advantages of both analytical approaches and neural networks. The basic idea is to start
with an analytical approach and an associated inference algorithm, and unfold the inference iterations as layers in a deep
network. After the size of the network is fixed, it is trained
to perform accurate inference. In [48], the unfolding framework is shown to be able to interpret conventional networks as
mean-field inference in Markov random fields, and obtain new
architectures by instead using belief propagation as the inference algorithm.
A similar approach is taken by Yang et al. [49], who designed
a DNN for reconstructing magnetic resonance images from
CS measurements. Each layer in their deep network graph is
explicitly implemented to mimic the step of the ADMM [50]
optimization procedure. The neural network parameters to be
learned include a nonlinear transformation of the CS measurements, the shrinkage function, the regularization function, in
addition to the various hyperparameters of the ADMM algorithm. All of these unknowns are optimized as a result of the
training of the neural network. Yang et al. [49] showed that
choosing a neural network in this way could achieve higher
reconstruction performance than state-of-the-art methods.
Some authors have also enforced a fixed operation with nontrainable weights within the network to influence the model to

operate in a particular way. For example, the encoder-decoder
CNN designed by Fischer et al. [32] for optical flow estimation uses a correlation layer at the end of the encoder part of
their network architecture. After computing two separate sets
of feature maps for each of the two input video frames, the
correlation layer explicitly computes the correlations between
the two sets. The result of this correlation is then used by the
decoder part of their network. Fischer et al. [32] hypothesize
that explicitly incorporating this correlation layer, instead of
having the network learn the operation, facilitates the learning process. Similarly, in their work on blind deconvolution,
Schuler et al. [51] perform end-to-end training of a deep layered architecture of which the first layers, corresponding to the
feature extraction step, are learned through training a CNN,
but the other two modules of the architecture are fixed and
correspond to operations implemented by traditional image
deconvolution. Their deep network learns to deblur the input
image by iteratively alternating between the three modules.

Using neural networks as denoisers in variable
splitting-based optimization methods
Recently, a new approach to directly combine analytical optimization methods with DNNs has been proposed. With the
use of variable splitting techniques, such as the ADMM and
the HQS methods [50], the inverse problem is split into two
subproblems: a fidelity term subproblem and a regularization
subproblem. The inverse problem is solved by alternating optimization. Recent research has proposed the use of DNNs to
tackle the regularization subproblem. More specifically, in the
context of the variable splitting methods, the regularization
step can be interpreted as a denoising procedure, in which the
restored image at a particular step of the algorithm is mapped
to a more plausible image with the guidance of the prior term.
Instead of hand-engineering the prior term, DNNs have been
recently proposed performing the regularization step. Zhang
et al. [52], e.g., show that their set of learned denoisers can be
incorporated into their optimization framework to solve other
problems besides image denoising, such as image deblurring
and image SR. The set of denoiser-CNNs essentially act as
prior terms that regularize the optimization-based restoration procedure for the inverse problem at hand. Similarly,
because Chang et al.'s [53] encoder-decoder CNN is trained
in an adversarial learning context (discussed in the section
"Using Generative Adversarial Networks to Learn Posteriors
for the Inverse Problem"), it acquires a prior knowledge that is
directly extracted from the statistics of the images seen in the
training data set, and not dependent on the type of the inverse
problem we are trying to solve. This allows the authors to apply
their trained model to other inverse problem tasks, such as CS,
image inpainting, and image SR.

Making careful design choices for solving inverse problems
Special caution must be taken when using CNNs for regression tasks. Architectural choices that may work for a classifier CNN may harm the learning process of a CNN trained
for solving an inverse problem using regression. For example,

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018