IEEE Signal Processing Magazine - January 2018 - 59

asses the empirical "symptoms" that might be experienced
theoretical insight has motivated research into cost functions
during training. These symptoms include:
based on alternative distances. Several of these are explored
■ difficulties in getting the pair of models to converge [5]
in the section "Alternative Formulations."
■ the generative model "collapsing" to generate very similar samples for different inputs [25]
Training tricks
■ the discriminator loss converging quickly to zero [26], proOne of the first major improvements in the training of GANs
viding no reliable path for gradient updates to the generator.
for generating images were the DCGAN architectures proSeveral authors suggested heuristic approaches to address
posed by Radford et al. [5]. This work was the result of an
these issues [1], [25]; these are discussed in the next section.
extensive exploration of CNN architectures previously used
Early attempts to explain why GAN training is unstable
in computer vision, and it resulted in a set of guidelines
were proposed by Goodfellow and Salifor constructing and training both the
mans et al. [1], [25], who observed that
generator and discriminator. In the section
The representations
gradient descent methods typically used
"Convolutional GANs," we alluded to the
that can be learned by
for updating both the parameters of the
importance of strided and fractionally stridGANs may be used in a
generator and discriminator are inaped convolutions [27], which are key compovariety of applications,
propriate when the solution to the optinents of the architectural design. This allows
mization problem posed by GAN training
including image synthesis, both the generator and the discriminator to
actually constitutes a saddle point. Salilearn good upsampling and downsampling
semantic image editing,
mans et al. provided a simple example
operations, which may contribute to improvestyle transfer, image
that shows this [25]. However, stochastic
ments in the quality of image synthesis. More
superresolution, and
gradient descent is often used to update
specifically to training, batch normalization
classification.
neural networks and there are well-devel[28] was recommended for use in both netoped machine-learning programming
works to stabilize training in deeper models.
environments that make it easy to construct and update netAnother suggestion was to minimize the number of fully conworks using stochastic gradient descent.
nected layers used to increase the feasibility of training deeper
Although an early theoretical treatment [1] showed that the
models. Finally, Radford et al. [5] showed that using leaky recgenerator is optimal when p g (x) = p data (x), a very neat result
tifying linear units (ReLUs) activation functions between the
with a strong underlying intuition, the real data samples reside
intermediate layers of the discriminator gave superior perforon a manifold that sits in a high-dimensional space of possible
mance over using regular ReLUs.
representations. For instance, if color image samples are of
Later, Salimans et al. [25] proposed further heuristic
size N # N # 3 with pixel values [0, R +] 3, the space that may
approaches for stabilizing the training of GANs. The first, feabe represented-which we can call X -is of dimensionality
ture matching, changes the objective of the generator slightly
3N 2, with each dimension taking values between zero and the
to increase the amount of information available. Specifically,
maximum measurable pixel intensity. The data samples in the
the discriminator is still trained to distinguish between real and
support of p data, however, constitute the manifold of the real
fake samples, but the generator is now trained to match the disdata associated with some particular problem, typically occucriminator's expected intermediate activations (features) of its
pying a very small part of the total space, X. Similarly, the
fake samples with the expected intermediate activations of the
samples produced by the generator should also occupy only a
real samples. The second, minibatch discrimination, adds an
small portion of X.
extra input to the discriminator, which is a feature that encodes
Arjovsky et al. [26] showed that the support p g (x) and
the distance between a given sample in a minibatch and the
p data (x) lie in a lower-dimensional space than that correspondother samples. This is intended to prevent mode collapse, as
ing to X. The consequence of this is that p g (x) and p data (x)
the discriminator can easily tell if the generator is producing
may have no overlap, and so there exists a nearly trivial disthe same outputs.
criminator that is capable of distinguishing real samples,
A third trick, heuristic averaging, penalizes the network
x~p data (x) from fake samples, x~p g (x) with 100% accuracy.
parameters if they deviate from a running average of previIn this case, the discriminator error quickly converges to zero.
ous values, which can help convergence to an equilibrium. The
Parameters of the generator may only be updated via the disfourth, virtual batch normalization, reduces the dependency
criminator, so when this happens, the gradients used for updatof one sample on the other samples in the minibatch by caling parameters of the generator also converge to zero and may
culating the batch statistics for normalization with the sample
no longer be useful for updates to the generator. Arjovsky et
placed within a reference minibatch that is fixed at the beginal.'s explanations account for several of the symptoms related
ning of training.
to GAN training [26].
Finally, one-sided label smoothing makes the target for the
Goodfellow et al. [1] also showed that when D is optimal,
discriminator 0.9 instead of one, smoothing the discriminator's
training G is equivalent to minimizing the Jensen-Shannon
classification boundary, hence preventing an overly confident
(JS) divergence between p g (x) and p data (x) . If D is not optidiscriminator that would provide weak gradients for the genmal, the update may be less meaningful or inaccurate. This
erator. Sønderby et al. [29] advanced the idea of challenging
IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018