IEEE Signal Processing Magazine - January 2018 - 33

It is fairly straightforward to adapt the optimization of the
models described in this article to a GAN setting. The model
g z (·) takes the place of the generator G. The discriminator D
is typically chosen to have the standard architecture of a classification CNN. Both parameters of G and D are optimized
simultaneously, through the use of the adversarial loss

model within a GAN framework. During training, the generator G learns to output HR images that look plausible to the eye
of the discriminator D.
The use of generative models to increase the quality of the
output images is not limited to the task of image SR. When
a generator G is trained to generate images from a random
vector z, the generator ultimately learns a transformation
l GAN = x~pE (x)[log (D (x))] + y~pE (y)[log (1 - (D (G (y))))], (4)
that maps a trivial distribution like a Gaussian distribution to
the complex distribution of natural images. We can use this
where D (x) is the label provided by the discriminator D
learned transformation for solving a CS task. Bora et al. [40]
when it receives a real image as input and D (G (y)) is the
show that given a measurement y that was obtained from a
label obtained when a synthesized image, G (y), is input to
random Gaussian measurement matrix A, we can estimate xt
D. This loss drives the discriminator network to correctly
by finding the optimal zt that minimizes AG (zt ) - y . Given
classify the samples as real or synthesized,
this optimal zt , we obtain the reconstrucand pushes the generator to synthesizing
tion through the mapping x = G (zt ) (We
With carefully designed
images that look real with the goal of foolchoices, we can guide our note, however, that solving the optimization
ing the discriminator [6]. Once the networks
above may be very difficult). While tradineural networks to learn
are trained, D is discarded and only G is
tional CS reconstruction methods make
operations similar to
used. In cGANs, the adversarial loss is usuassumptions regarding the structure of natthose implemented by
ally optimized in addition to the pixel-wise
ural images, the approach described here
analytical models.
MSE loss. Similarly to the loss in feature
requires no such assumption regarding the
space described in the section "Training
structure of the image. Instead, all prior
Procedure of DNNs for Inverse Problems," the adversarial loss
knowledge needed to reconstruct the image was indirectly
acts as a regularizer to the negative perceptual effects that may
learned from a large training data set.
result from optimizing the pixel-wise MSE. We note here that
the generator G does not explicitly model the distribution of
Variational autoencoders for inverse problems
images p (x | y), but instead implicitly models it through its inAlthough we have only talked about GANs as generative modteraction with the discriminator during training. The cGANs'
els, another type of neural networks, variational autoencoders
ability to indirectly learn the complex distribution of natural
(VAEs), have also been successful in capturing the complex
image densities has shown to significantly increase the quality
distribution of natural images. VAEs are composed of an
of images generated for SR (see [24], [25], and [55]) and image
encoder and a decoder. The encoder outputs the parameters
inpainting (see [31]).
of the distribution of the latent variable z given either y or x
depending on the problem, i.e., the encoder learns a conditional
Solving SR and CS problems in an unsupervised context
distribution. Given this distribution, we can sample a random
In this article, we have, until now, focused on solving an inverse
vector z and pass it through the decoder part of the network to
problem as a regression problem, which required the use of a
output an image that looks like it is drawn from the distribution
training data set with example pairs (y, x). In this section, we
of natural images. In their work on learning representations
discuss some of the methods that solve the SR and CS problems
for CS, Bora et al. [40] experiment with the use of the decoder
following an unsupervised learning approach.
part of the VAE to map z to an image x = G (z). While VAEs,
Sønderby et al. [55] reformulate the SR problem as a
just like GANs, provide a way to produce an output image that
maximum a posteriori estimation problem. They show that,
looks natural, they have been less popular than GANs in their
by imposing several architectural constraints on their CNNuse for solving inverse problems in imaging.
based model, the range of functions learned by the model are
restricted to valid SR function only. To achieve this, they use
Limitations of the use of neural networks
their knowledge of the downsampling operation T used for SR
for inverse problems in imaging
and define a parametric function class that guarantees that the
output of their model x, given y as input, is consistent with
The knowledge of neural networks is constrained
the forward downsampling modeling y = Tx. By restricting
to the data seen during training
the set of functions in this way, the knowledge of the relationThe functional relationship between input and output of the
ship between x and y is explicitly wired into the architecturmodel are highly based on the image pairs (y, x) seen dural design of the network. This result is quite powerful, as it
ing training. We usually do not have access to a data set that
allows us to depart from the traditional supervised approach
contains labeled real-world images x and their corresponding
and instead use unsupervised generative methods for solving
transformations y, and therefore we have to resort to artifithe SR tasks with DNNs. Approaching the task as a generative
cially generating the data set. There are multiple issues with
one, the authors train their neural network architecture to outthis approach. First, the neural network reconstruction ability
put HR images of high perceptual quality by optimizing their
will be highly dependent on the choice of T used to create the
data

data

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018