Signal Processing - November 2017 - 93

Algorithm descriptions and reproducibility
When planning this survey, we aimed to measure quantitative
trends in the literature, e.g., to plot the number of training samples
versus the number of parameters for each network. We quickly
discovered this is nearly impossible. Very few manuscripts clearly noted the number of parameters they were training, and only
some provided a clear-enough description of the network for us
to calculate the value. Many more included a figure of network
architecture along the lines of Figure 2, but without a clear statement of the dimensions of each layer. Similar problems exist in
the description of the training and evaluation procedures, where
it is not always clear whether the evaluation data come from
simulation or from a real data set. As the field matures, we hope
papers converge on a standard way to describe network architecture, training, and evaluation.
The lack of clarity presents a barrier to the reproducibility of the work. Another barrier is the fact that training often
requires specialized or expensive hardware. While GPUs have
become more ubiquitous, the largest (and best-performing)
CNNs remain difficult for small research groups to train. For
example, the CNN that won the ImageNet Large-Scale Visual
Recognition Challenge in 2012 took "between five and six
days to train on two GTX 580 3GB GPU" [5].

Robustness of learning
The success of any CNN-based algorithm hinges on finding a
reasonable solution to the learning problem (3). As stated previously, this is a nonconvex problem, where the best solution
we can hope for is to find one of many local minima of the
cost. This raises questions about the robustness of the learning
to changes in the initialization of parameters and the specifics
of the optimization method employed. This is in contrast to
the typical convex formulations of inverse problems, where the
specifics of the initialization and optimization scheme provably do not affect the quality of the result.
The uncertainty about learning complicates the comparison
of any two CNN-based methods. Does A outperform B because
of its superior architecture, or simply because the optimization
of A fell into a superior local minimum? As an example of the
confusion this can cause, [34] shows, in the context of denoising,
superresolution, and JPEG deblocking that a network trained
with the l 1 cost function can outperform a network trained
with the l 2 cost function even with regard to the l 2 cost. In the
authors' analysis of this highly disturbing result, they attribute it
to the l 2 learning being stuck in a local optimum. Regard-less,
the vast majority of work relies on the l 2 cost, which is computationally convenient and provides excellent results.
There is some indication that large networks trained with
lots of data can overcome this problem. In [40], the authors
show that larger networks have more local minima, but that
most local minima are equivalent in terms of testing performance. They also identify that the global minima likely correspond to parameter settings that overfit the training set. More
work on the stability of the learning process will be an important step toward wider acceptance of CNNs in the inverse
problem community.

More generally, how sensitive are the results of a given experiment to small changes in the training set, network architecture,
or optimization procedure? Is it possible for the experimenter to
overfit the testing set by iteratively tweaking the network architecture (or the experimental parameters) until state-of-the-art
results are achieved? To combat this, CNN-based approaches
should provide carefully constructed experiments with results
reported on a large number of testing images. Even better are
competition data sets, where the testing data is hidden until algorithm development is complete.

Can we trust the results?
Once trained, CNNs remain nonlinear and highly complex.
Can we trust reconstructions generated by such systems?
One way to look at this is to evaluate the sensitivity of the
network to noise: ideally, small changes to the input should
cause only small changes to the output; data augmentation
during training can help achieve this. Similarly, demonstrating generalization between data sets (where the network
learns on one data set, but is evaluated on another) can help
improve confidence in the results by showing that the performance of the network is not dependent on some systematic
bias of the data set.
A related question is how to measure the quality of the results.
Even if a robust SNR improvement can be demonstrated, practitioners will inevitably want to know, e.g., whether the resulting images can be reliably used for diagnosis. To this end, as
much as possible, methods should be assessed with respect to the
ultimate application of the reconstruction (diagnosis, quantification of biological phenomenon, etc.) rather than an intermediate
measure such as SNR or structural similarity (SSIM). While this
critique can be made of any approach to inverse problems, it is
especially relevant for CNNs because they are often treated as
black boxes and because the reconstructions they generate are
plausible-looking by design, hiding areas where the algorithm is
less sure of the result.

Next steps
So far, we have given a small look into the wide variety of ways
that researchers have applied CNNs to solve inverse problems
in imaging. Because CNNs are so powerful and flexible, we
believe there is plenty of room to create even better systems.
Next, we suggest a few directions that this future research
might take.

Biomedical imaging
CNNs have so far been applied mostly to inverse problems where
the measurements take the form of an image and the measurement
model is simple, and less so for CT and MRI, which have relatively more complicated models. A search on arXiv.org reveals
dozens more CT and MRI papers submitted within the last few
months, suggesting many incoming contributions in these areas.
We expect diffusion into other modalities such as positron-emission tomography, single-photon emission CT, transmission electron microscopy, structured illumination microscopy, ultrasound,
superresolution microscopy, etc. to follow.

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

http://www.arXiv.org

Table of Contents for the Digital Edition of Signal Processing - November 2017