Signal Processing - November 2017 - 86

networks and, specifically, CNNs-not only for computer vision
tasks but also for inverse problems.
The purpose of this article is to summarize the recent works
using CNNs for inverse problems in imaging, i.e., in problems
most naturally formulated as recovering an image from a set
of noisy measurements. This criterion excludes detection, segmentation, classification, quality assessment, etc. We also focus
on CNNs, avoiding other architectures such as recurrent neural networks, fully connected networks, and stacked denoising
autoencoders. We organized our literature search by application,
selecting topics of broad interest where we could find at least
three peer-reviewed papers from the last ten years. (Much of the
work on the theory and practice of CNNs is posted on the preprint server arXiv.org before eventually appearing in traditional
journals. Because of the lack of peer review on arXiv.org, we
have preferred not to cite these papers, except in cases where we
are trying to illustrate a very recent trend or future direction for
the field.) The resulting applications and references are summarized in Table 1. The aim of this constrained scope is to allow
us to draw meaningful generalizations from the surveyed works.

Background
We begin by introducing inverse problems and contrasting the
traditional approach to solving them with a learning-based
approach. For a textbook treatment of inverse problems, see
[28]. Throughout the section, we use X-ray computed tomograTable 1. Reviewed applications and associated references.
Denoising

Deconvolution

Superresolution

[6]-[11]

[10], [12]-[14] [9], [15]-[20]

phy (CT) as a running example, and Figure 1 shows images of
the various mathematical quantities we mention.

Learning for inverse problems in imaging
Mathematically speaking, an imaging system is an operator
H : X " Y that acts on an image x ! X, to create a vector of
measurements y ! Y, with H {x} = y. The underlying function/vector spaces are
■■ the space, X, of acceptable images, which can be twodimensional (2-D), three-dimensional (3-D), or even
3-D+time, with its values representing a physical quantity
of interest, such as X-ray attenuation or concentration of
fluorophores
■■ the space, Y, of measurement vectors that depends on the
imaging operator and could include images (discrete arrays
of pixels), Fourier samples, line integrals, etc.
We typically consider x to be a continuous object (function of
space), while y is usually discrete: Y = R M. For example, in
X-ray CT, x is an image representing X-ray attenuations, H represents the physics of the X-ray source and detector, and y is the
measured sinogram (see Figure 1).
In an inverse imaging problem, we aim to develop a reconstruction algorithm (which is also an operator), R: Y " X,
to recover the original image, x, from the measurements, y.
The dominant approach for reconstruction, which we call the
objective function approach, is to model H and recover an
estimate of x from y by
R obj " y , = argmin f ^ H " x ,, y h, (1)

	

MRI

CT

[21]-[23]

[24]-[27]

x!X

where H: X " Y is the system model, which is usually linear, and f : Y # Y " R + is an appropriate measure of error.

CNNθ
?
∼
H -1

HT

Rreg

y

H
∼
H -1 {y }

∼
HT {y }
CNNθ

x∼
CNNθ

CNNθ
[25], [27]

?

[25]

x

FIGURE 1. A block diagram of image reconstruction methods, using images from X-ray CT as examples. An image, x, creates measurements, y, that can

be used to estimate x in a variety of ways. The traditional approach is to apply a direct inversion, Hu -1, which is artifact prone in the sparse-measurement
case (note the stripes in the reconstruction). The current state of the art is a regularized reconstruction, R reg, written, in general, in (2). Several recent
works apply CNNs to the result of the direct inversion or an iterative reconstruction, but it might also be reasonable to use as input the measurements
themselves or the back projected measurements.

86

IEEE SIGNAL PROCESSING MAGAZINE

|

November 2017

|
http://www.arXiv.org http://www.arXiv.org
Table of Contents for the Digital Edition of Signal Processing - November 2017
