IEEE Signal Processing Magazine - January 2018 - 60

the discriminator by adding noise to the
What sets GANs apart from function derived for the WGAN relies on
samples before feeding them into the disthe discriminator, which they refer to as
these standard tools of
criminator. Sønderby et al. [29] argued that
the critic, being a k-Lipschitz continuous
signal processing is the
one-sided label smoothing biases the optifunction; practically, this may be implelevel of complexity of the
mal discriminator, while their technique,
mented by simply clipping the parameters
models that map vectors
instance noise, moves the manifolds of the
of the discriminator. However, more recent
real and fake samples closer together, at the
research [33] suggested that weight clipping
from latent space to
same time preventing the discriminator easadversely reduces the capacity of the disimage space.
ily finding a discrimination boundary that
criminator model, forcing it to learn simpler
completely separates the real and fake samfunctions. Gulrajani et al. [33] proposed an
ples. In practice, this can be implemented by adding Gaussian
improved method for training the discriminator for a WGAN,
noise to both the synthesized and real images, annealing the
by penalizing the norm of discriminator gradients with respect
standard deviation over time. The same process was indepento data samples during training, rather than performing paramdently proposed by Arjovsky et al. [26].
eter clipping.

Alternative formulations

A brief comparison of GAN variants

The first part of this section considers other information-theoretic interpretations and generalizations of GANs. The second part looks at alternative cost functions that aim to directly
address the problem of vanishing gradients.

GANs allow us to synthesize novel data samples from random
noise, but they are considered difficult to train due partially to
vanishing gradients. All GAN models that we have discussed
in this article require careful hyperparameter tuning and model
selection for training. However, perhaps the easier models to
train are the adversarial autoencoder (AAE) and the WGAN.
The AAE is relatively easy to train because the adversarial loss
is applied to a fairly simple distribution in lower dimensions
(than the image data). The WGAN [33], is designed to be easier
to train, using a different formulation of the training objective
that does not suffer from the vanishing gradient problem. The
WGAN may also be trained successfully even without batch
normalization; it is also less sensitive to the choice of nonlinearities used between convolutional layers.
Samples synthesized using a GAN or WGAN may belong
to any class present in the training data. Conditional GANs
provide an approach to synthesizing samples with userspecified content.
It is evident from various visualization techniques (Figure 6) that the organization of the latent space harbors some
meaning, but vanilla GANs do not provide an inference
model to allow data samples to be mapped to latent representations. Both BiGANs and ALI provide a mechanism
to map image data to a latent space (inference), however,
reconstruction quality suggests that they do not necessarily
faithfully encode and decode samples. A very recent development shows that ALI may recover encoded data samples
faithfully [21]. However, this model shares a lot in common
with the AVB and AAE. These are autoencoders, similar
to VAEs, where the latent space is regularized using adversarial training rather than a KL-divergence between encoded
samples and a prior.

Generalizations of the GAN cost function
Nowozin et al. [30] showed that GAN training may be generalized to minimize not only the JS divergence, but an estimate
of f-divergences; these are referred to as f-GANs. The f-divergences include well-known divergence measures such as the
KL-divergence. Nowozin et al. showed that the f-divergence may
be approximated by applying the Fenchel conjugates of the
desired f-divergence to samples drawn from the distribution of
generated samples, after passing those samples through a discriminator [30]. They provide a list of Fenchel conjugates for
commonly used f-divergences, as well as activation functions
that may be used in the final layer of the generator network,
depending on the choice of f-divergence. Having derived the
generalized cost functions for training the generator and discriminator of an f-GAN, Nowozin et al. [30] observe that, in
its raw form, maximizing the generator objective is likely to
lead to weak gradients, especially at the start of training, and
proposed an alternative cost function for updating the generator, which is less likely to saturate at the beginning of training.
Nowozin et al. proposed that when the discriminator is trained,
the derivative of the f-divergence on the ratio of the real and
fake data distributions is estimated, while when the generator
is trained only an estimate of the f-divergence is minimized.
Uehara et al. [31] extend the f-GAN further, where in the discriminator step the ratio of the distributions of real and fake
data are predicted, and in the generator step the f-divergence is
directly minimized. Alternatives to the JS-divergence are also
covered by Goodfellow [12].

Alternative cost functions to prevent vanishing gradients
Arjovsky et al. [32] proposed the Wasserstein GAN (WGAN),
a GAN with an alternative cost function that is derived from an
approximation of the Wasserstein distance. Unlike the original GAN cost function, the WGAN is more likely to provide
gradients that are useful for updating the generator. The cost
60

The structure of latent space
GANs build their own representations of the data they are
trained on, and in doing so produce structured geometric vector spaces for different domains. This is a quality shared with
other neural network models, including VAEs [23], as well
as linguistic models such as word2vec [34]. In general, the
domain of the data to be modeled is mapped to a vector space,

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018