IEEE Signal Processing Magazine - January 2018 - 35

task at hand, and may be useful for refining an initial solution
provided by an analytical method. Encoder-decoder CNNs
with skip connections explained in the section "Learning Higher-Level Representations with Encoder-Decoder
CNNs," have also provided our deep-learning models with
new ways to reconstruct images from given observations. In
addition to more intricate architectures, more appropriate
loss functions were recently proposed, such as the perceptual loss function in feature space described in the section
"Training Procedure of DNNs for Inverse Problems" or the
unsupervised GAN loss function introduced in the section
"Using Generative Adversarial Networks to Learn Posteriors
for the Inverse Problem," which paves the way for the use of
powerful statistical models in the near future. In addition to
using DNNs in a regression framework, many methods have
made the choice to explicitly train neural networks for what
they are known to do best-feature extraction (see the section
"Training Deep Neural Networks to Learn New Representations of Natural Images"). In particular, the use of CNNs to
extract the statistics of natural images and use these learned
representations to solve an inverse problem is a promising
approach that needs to be further explored. This approach
has already been investigated and proven successful in the
context of variable splitting methods, autoencoders, or with
generative approaches.
While the techniques described in this article are successful at solving inverse problems in imaging, sometimes
surpassing the analytical state of the art, the challenge of
bringing the large amount of prior knowledge we possess as
engineers into the deep-learning framework still remains. The
most critical issue in today's research lies in the fact that we
are, in most of the techniques described previously, essentially using a "black-box" model for solving a problem for
which we possess a considerable quantity of knowledge and
understanding. Therefore, more research reframing the use
of DNNs in a context in which we can apply some of our
domain-based knowledge is needed to simultaneously benefit
from the advantages of deep-learning and analytical methods when solving inverse problems in imaging. In the future,
this research may manifest itself in terms of the introduction
of new layers and operations in the deep-learning system
that are specifically tailored to the inverse problem at hand.
Moreover, as we aim to combine deep-learning and analytical
methods, we expect that future research will depart from the
traditional end-to-end mapping approach and instead focus
on solving a very specific step of the formulated inverse problem. Finally, we will undoubtedly see more research in the
coming years on the use of generative models to solve image
recovery tasks, as has already been manifested with the recent
introduction of GANs toward solving various inverse problems. These different future directions have in common a key
challenge that remains to be addressed, that of achieving the
optimal balance between imposing engineering knowledge
into the framework, and, simultaneously, making use of the
ever-growing potential of deep learning to solve problems for
which we do not have answers.

Acknowledgments
This work was supported in part by the U.S. Department of Energy under grant DE-NA0002520, Office of Naval Research award
N00014-15-1-2735, National Science Foundation IDEAS program, the Defense Advanced Research Projects Agency ReImagine, and the Spanish Ministry of Economy and Competitiveness
through projects TIN2013-43880-R and DPI2016-77869-C2-2-R.

Authors
Alice Lucas (alicelucas2015@u.northwestern.edu) received her
B.S. degree in applied math, engineering, and physics from the
University of Wisconsin-Madison in 2015. She is currently pursuing her Ph.D. degree in the Electrical Engineering and
Computer Science Department at Northwestern University,
Evanston, Illinois. Her research at the Image and Video
Processing Laboratory involves the use of deep-learning models
for various image processing tasks, currently focusing on the
task of video superresolution.
Michael Iliadis (michaililiadis2016@u.northwestern.edu) received his B.S. degree in digital systems from the University of
Piraeus, Greece, in 2008, his M.S. degree in computer science
from the University of Bath, United Kingdom, in 2009, and his
Ph.D. degree in computer science from Northwestern University,
Evanston, Illinois in 2016. He spent the summers of 2013 and
2014 at TCL Research America as a research intern. In 2016, he
joined Sony U.S. Research Center, San Jose, California, where
he is currently a research scientist. He holds five U.S. patents.
His current research interests include machine (deep) learning,
sparse modeling, compressive sensing, superresolution, face
recognition, and semantic segmentation.
Rafael Molina (rms@decsai.ugr.es) received his B.Sc.
degree in mathematics (statistics) and his Ph.D. degree in optimal design in linear models from the University of Granada,
Spain, in 1979 and 1983, respectively. His research interest
focuses mainly on using Bayesian modeling and inference in
problems, such as image restoration (applications to astronomy and medicine), superresolution of images and video, blind
deconvolution, computational photography, source recovery in
medicine, compressive sensing, low-rank matrix decomposition, active learning, fusion, and classification. He is currently
a professor of computer science and artificial intelligence at
the University of Granada.
Aggelos K. Katsaggelos (aggk@eecs.northwestern.edu)
received his diploma degree in electrical and mechanical engineering in 1979 from the Aristotelian University in Thessaloniki,
Greece, and his M.S. and Ph.D. degrees, both in electrical engineering, from the Georgia Institute of Technology, Atlanta, in
1981 and 1985, respectively. He is currently a professor of electrical engineering and computer science and holds the Joseph
Cummings chair in the Department of Electrical Engineering
and Computer Science at Northwestern University, Evanston,
Illinois. He has published seven books, more than 250 journal
papers, 500 conference papers, and 40 book chapters in the
areas of multimedia signal processing and communications,
computational imaging, and machine learning. He holds 26
international patents.

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018