IEEE Signal Processing Magazine - January 2018 - 61

which has fewer dimensions than the data
vised learning. Image synthesis remains a
GANs build their own
space, forcing the model to discover incore GAN capability and is especially userepresentations of the
teresting structure in the data and repreful when the generated image can be subject
data they are trained on,
sent it efficiently. This latent space is
to pre-existing constraints. Superresolution
and in doing so produce
at the "originating" end of the generator
[36]-[38] offers an example of how an existstructured geometric
network, and the data at this level of reping approach can be supplemented with an
resentation (the latent space) can be highly
adversarial loss component to achieve highvector spaces for
structured and may support high-level seer-quality results. Finally, image-to-image
different domains.
mantic operations [5]. Examples include
translation demonstrates how GANs offer a
the rotation of faces from trajectories
general-purpose solution to a family of tasks
through latent space, as well as image analogies that have
that require automatically converting an input image into an
the effect of adding visual attributes such as eyeglasses onto
output image.
a "bare" face.
All (vanilla) GAN models have a generator that maps data
Classification and regression
from the latent space into the space to be modeled, but many
After GAN training is complete, the neural network can be
GAN models have an encoder that additionally supports the
reused for other downstream tasks. For example, outputs of
inverse mapping [19], [20]. This becomes a powerful meththe convolutional layers of the discriminator can be used as
od for exploring and using the structured latent space of the
a feature extractor, with simple linear models fitted on top of
GAN network. With an encoder, collections of labeled images
these features using a modest quantity of (image, label) pairs
can be mapped into latent spaces and analyzed to discover
[5], [25]. The quality of the unsupervised representations with"concept vectors" that represent high-level attributes such as
in a DCGAN network have been assessed by applying a regu"smiling" or "wearing a hat." These vectors can be applied at
larized L2-SVM classifier to a feature vector extracted from
scaled offsets in latent space to influence the behavior of the
the (trained) discriminator [5]. Good classification scores were
generator (Figure 6). Similar to using an encoding process to
achieved using this approach on both supervised and semisumodel the distribution of latent samples, Gurumurthy et al.
pervised data sets, even those that were disjoint from the origi[35] propose modeling the latent space as a mixture of Gaussnal training data.
ians and learning the mixture components that maximize the
The quality of the data representation may be improved
likelihood of generated data samples under the data generatwhen adversarial training includes jointly learning an infering distribution.
ence mechanism such as with ALI [19]. A representation vector was built using last three hidden layers of the ALI encoder,
Applications of GANs
a similar L2-SVM classifier, yet achieved a misclassification
Discovering new applications for adversarial training of deep
rate significantly lower than the DCGAN [19]. Additionally,
networks is an active area of research. We examine a few
ALI has achieved state-of-the art classification results when
computer vision applications that have appeared in the literalabel information is incorporated into the training routine.
ture and been subsequently refined. These applications were
When labeled training data is in limited supply, adversarial
chosen to highlight some different approaches to using GANtraining may also be used to synthesize more training samples.
based representations for image manipulation, analysis, or
Shrivastava et al. [39] use GANs to refine synthetic images
characterization and do not fully reflect the potential breadth
while maintaining their annotation information. By training
of application of GANs.
models only on GAN-refined synthetic images (i.e., no real
Using GANs for image classification places them within
training data) Shrivastava et al. [39] achieved state-of-the-art
the broader context of machine learning and provides a useful
performance on pose- and gaze-estimation tasks. Similarly,
quantitative assessment of the features extracted in unsupergood results were obtained for gaze estimation and prediction

FIGURE 6. An example of applying a "smile vector" with an ALI model [19]. The first image is an example of an unsmiling woman and the last is an
example of a woman smiling. A z value for the first image is inferred, z 1 and for the last, z 2. Interpolating along a vector that connects z 1 and z 2, gives
z values that may be passed through a generator to synthesize novel samples. Note the implication: a displacement vector in latent space traverses smile
"intensity" in image space. (Figure used courtesy of Tom White.)
IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018