Signal Processing - November 2017 - 94

Central to this work will be questions of how best to combine CNNs with knowledge of the underlying physics as well as
direct and iterative inversion techniques. Most of the surveyed
works involve using a CNN to correct the artifacts created by
direct or iterative methods, where it remains an open question
what is the best such prereconstruction method. One creative
approach is to build the inverse operator into the network architecture as in [22], where the network can compute inverse Fourier transforms. Another would be to use the back-projected
measurements, H T y, which at least take the form of an image
and could reduce the burden on the CNN to learn the physics
of the forward model. CNNs could be deployed in a variety of
other ways here, too, e.g. using a CNN to approximate a high
quality, but extremely slow reconstruction method. With enough
computing power, a training set could be generated by running
the slow method on real data, and, once trained, the resulting
network could provide very fast and accurate reconstructions.

Cross-task learning
In cross-task learning (also called transfer learning, although
this can have other meanings as well), an algorithm is trained
with one data set and deployed on a different, but related,
task. This is attractive in the inverse problem setting because
it avoids the costly retraining of the network when imaging
parameters change (different noise levels, image dimensions,
etc.), which may occur often. Or we could imagine a network
that transfers between completely different imaging modalities, especially when training data for the target modality are
scarce; e.g., a network could train on denoising natural images
and then be used to reconstruct MRI images. Recent work has
made progress in this direction by learning a CNN-based proximal operator, which can be used inside an iterative optimization method for any inverse problem [32].

Multidimensional signals
Modern inverse problems in imaging increasingly involve
reconstruction of 3-D or 3-D+time images. However, most
CNN-based approaches to these problems involve 2-D inputs
and outputs. This is likely because much of the work on deep
neural networks in general has been in two dimensions and
because of practical considerations. Specifically, learning
strongly relies on GPU computation, but current GPUs have
maximally 24 GB of physical memory. This limitation makes
training a large network with 3-D inputs and outputs infeasible.
One way to overcome this issue is model parallelism, in
which a large model is partitioned onto separable computers.
Another is data parallelism, where it is the data that are split.
When used together, large computational gains are achieved
[41]. Such approaches will be key in tackling multidimensional
imaging problems.

Generative adversarial networks and perceptual loss
CNN-based approaches to inverse problems also stand to benefit from new developments in neural network research. One
such development is the generative adversarial network (GAN)
[42], which may offer a way to break current limits in supervised
94

learning. Basically, two networks are trained in competition:
the generator tries to learn a mapping between training samples, while the discriminator attempts to distinguish between
the output of the generator and real data. Such a setup can,
e.g., produce a generator capable of creating plausible natural
images from noise. The GAN essentially revises the learning
formulation (3) by replacing the cost function f with another
neural network. In contrast to a designed cost function, which
will be suboptimal if the assumed noise model is incorrect, the
discriminator network learns a cost function that models the
probability density of the real data. GANs have already begun
to be used for inverse problems, e.g., for superresolution in [30]
and deblurring in [14].
A related approach is perceptual loss, where a network is
trained to compute a loss function that matches human perception. The method has already been used for style transfer
and superresolution [43]. Compared to the standard Euclidean
loss, networks trained with perceptual loss give better looking
results, but do not typically improve the SNR. It remains to be
seen whether these ideas can gain acceptance for applications
such as medical imaging, where the results must be quantitatively accurate.

Acknowledgment
As stated in the introduction, [30], [32], and [35] from arXiv.org
are not peer reviewed. They have been included only to illustrate
recent trends.

Authors
Michael T. McCann (michael.mccann@epfl.ch) received his
B.S.E. degree in biomedical engineering in 2010 from the
University of Michigan and the Ph.D. degree in biomedical
engineering from Carnegie Mellon University, Pittsburgh,
Pennsylvania, in 2015. He is currently a scientist with the
Laboratoire d'imagerie biomédicale and Centre d'imagerie
biomédicale, École Polytechnique Fédérale de Lausanne,
where he works on X-ray computed tomography reconstruction. His research interest centers on developing signal
-processing tools to answer biomedical questions.
Kyong Hwan Jin (kyong.jin@epfl.ch) received his B.S.
degree and integrated M.S. and Ph.D. degrees from the
Department of Bio and Brain Engineering at Korea Advanced
Institute of Science and Technology (KAIST), Daejeon, in
2008 and 2015, respectively. He was a postdoctoral scholar at
KAIST from 2015 to 2016. He is currently a postdoctoral
scholar in the Biomedical Imaging group, École Polytechnique
Fédérale de Lausanne, Switzerland. His research interests
include low-rank matrix completion, sparsity-promoted signal
recovery, sampling theory, biomedical imaging, and image processing in various applications.
Michael Unser (michael.unser@epfl.ch) received the M.S.
and Ph.D. degrees in electrical engineering in 1981 and 1984,
respectively, from the École Polytechnique Fédérale de
Lausanne (EPFL). He is a professor and the director of the
EPFL Biomedical Imaging Group, Lausanne, Switzerland. His
primary area of investigation is biomedical image processing.

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

http://www.arXiv.org

Table of Contents for the Digital Edition of Signal Processing - November 2017