IEEE Signal Processing Magazine - January 2018 - 54

Synthetic
Data
Sample

x′

The generator is trained
to map a noise sample
to a synthetic data sample
that can "fool"
the discriminator.

Real
or
Fake?

Noise
Source

Real Data
Sample

The discriminator is trained to
distinguish real data
samples from synthesized
samples.

FIGURE 1. The two models that are learned during the training process for a GAN are the discriminator ^D h and the generator ^Gh. These are typically

implemented with neural networks, but they could be implemented by any form of differentiable system that maps data from one space to another; see
article text for details.

of convolutional and/or fully connected layers. The generator
and discriminator networks must be differentiable, though it is
not necessary for them to be directly invertible. If one considers the generator network as mapping from some representation space, called a latent space, to the space of the data
(we shall focus on images), then we may express this more formally as G: G (z) " R x , where z ! R | z | is a sample from the
latent space, x ! R | x | is an image and · denotes the number
of dimensions.
In a basic GAN, the discriminator network, D, may be similarly characterized as a function that maps from image data to
a probability that the image is from the real data distribution,
rather than the generator distribution: D : D (x) " (0, 1) . For
a fixed generator, G, the discriminator, D, may be trained to
classify images as either being from the training data (real,
close to one) or from a fixed generator (fake, close to zero).
When the discriminator is optimal, it may be frozen, and the
generator, G, may continue to be trained so as to lower the
accuracy of the discriminator. If the generator distribution is
able to match the real data distribution perfectly, then the discriminator will be maximally confused, predicting 0.5 for all
inputs. In practice, the discriminator might not be trained until
it is optimal; we explore the training process in more depth in
the section "Training GANs."
On top of the interesting academic problems related to
training and constructing GANs, the motivations behind training GANs may not necessarily be the generator or the discriminator per se: the representations embodied by either of the pair
of networks can be used in a variety of subsequent tasks. We
explore the applications of these representations in the section
"Application of GANs."
54

Preliminaries
Terminology
Generative models learn to capture the statistical distribution
of training data, allowing us to synthesize samples from the
learned distribution. On top of synthesizing novel data samples, which may be used for downstream tasks such as semantic image editing [2], data augmentation [3], and style transfer
[4], we are also interested in using the representations that
such models learn for tasks such as classification [5] and
image retrieval [6].
We occasionally refer to fully connected and convolutional
layers of deep networks; these are generalizations of perceptrons or spatial filter banks with nonlinear postprocessing.
In all cases, the network weights are learned through
backpropagation [7].

Notation
The GAN literature generally deals with multidimensional vectors and often represents vectors in a probability space by italics (e.g., latent space is z) . In the field of signal processing, it is
common to represent vectors by bold, lowercase symbols, and
we adopt this convention to emphasize the multidimensional
nature of variables. Accordingly, we will commonly refer to
p data (x) as representing the probability density function over a
random vector x that lies in R | x | . We will use p g (x) to denote
the distribution of the vectors produced by the generator network of the GAN. We use the calligraphic symbols G and D
to denote the generator and discriminator networks, respectively. Both networks have sets of parameters (weights), H D
and H G , that are learned through optimization, during training.

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018