IEEE Circuits and Systems Magazine - Q4 2019 - 32

image from G. The output of D is a value in the range [0,1]
presenting the probability of whether the input is from
the real image pdata or generated image pg. The feedback
from D will affect the training of G to prompt the generation of images with higher fidelity, which will be fed to
D for further examination. On the contrary, the discriminator is expected to distinguish 'fake' images from real
images. This adversarial training procedure, as shown in
Fig. 14, is repeated until a balance is reached between the
generator G and the discriminator D.
Generally, CNNs are assigned as the primary network
for the discriminator to do the image classification. The
generator is, in a sense, a type of decoder: taking a vector of random noise and upsampling it to an image. The
networks are each optimizing an individual but opposing objective function. As a minimax game between D
and G, the value function V^G, D h is given below:
minmax V ^ D, G h = E x~p data ^ x h 6log D ^ x h@
G

+ E z~p z ^ z h 6log ^1 - D ^G ^ z hhh@

(5)

In practice, rather than training G to minimize
log ^1-D ^G ^ z hhh, we always maximize log D^G^ z hh to supply sufficient gradients early in the training [43].
B. Wasserstein GAN: A Vital Improvement
Although GANs have shown great success in various
applications, the training of a practicable GAN is a
tricky process. This process is well known to be slow
and unstable [116]. Some of the common problems are
listed below:
■ Unstable training: In the GAN's training procedure, two adversarial models are trained simultaneously. However, the connection between the
costs of these two models is independent. Updating the gradient of both models at the same time
cannot guarantee a convergence. Moreover, if the
performance of the discriminator is too perfect
the loss function (5) will fall to zero, which results
in no gradient to update. Hence it is a dilemma
that the discriminator needs to behave neither
too badly nor too well.
■ Mode collapse: During the training, researchers
have found that sometimes the generated images from G are in the same pattern [117]. In other
words, the generator is short of sufficient diversity that the 'fake' images created look totally
the same. This problem is referred to as mode
collapse, a common failure scenario for practical
applications of GANs. The generator trapped by
mode collapse fails to learn the representation of
the real-world data distribution and keeps creating images with low variety.
32

IEEE CIRCUITS AND SYSTEMS MAGAZINE

■ Lack of an evaluation metric: The objection func-

tion of GANs cannot give a clear indication of the
training progress. The relationship between the
training loss curves and the training progress is
confused and hard to be interpreted. There is also
no explicit indicator as to the stopping criterion of
the training process.
To address the above problems, many possible solutions have been discussed and analyzed and to date,
the best solution is provided by the Wasserstein GAN
(WGAN) [118]. In the WGAN, the Wasserstein distance
is introduced to replace the conventional metrics,
i.e. Kullback-Leibler (KL) or Jensen-Shannon (JS) divergence, for quantifying the similarity between two
probability distributions. Compared to the KL and JS
divergence, the Wasserstein distance can reflect the
differential between two distributions even if there is
no overlapping part. Only the Wasserstein metric keeps
a smooth measure for a stable learning process using
gradient descents.
The modified loss functions of the generator and the
discriminator in a WGAN are L ^G h = - E x~p g 6D ^ x h@ and
L ^ D h = E x~p g 6D ^ x h@ - E x~p data 6D ^ x h@, respectively. According to the loss functions, the discriminator is trained
to learn a K-Lipschitz continuous function to compute
the Wasserstein distance, instead of directly telling the
fake samples apart from the real ones. A smaller Wasserstein distance means the output of the generator is
closer to the real data distribution. For maintaining the
K-Lipschitz continuity of D, weights clipping is applied
after updating every gradient to enforce a Lipschitz
constraint.
In [119], a gradient penalty is adopted to solve the
problems of exploding gradients and vanishing gradients caused by weights clipping in the original WGAN.
The new model is named the WGAN-gradient penalty
(WGAN-GP). The WGAN-GP has been shown to perform better than the standard WGAN and enables stable
training of a wide variety of GAN models. Almost no hyper-parameter tuning is required however the time for
training is significantly increased.
Some other remarkable variants of the vanilla GAN
include:
■ Deep Convolutional GAN (DCGAN) [105]: This is
a popular CNN-based GAN, which combines the
CNNs in supervised learning and the GANs in unsupervised learning. A set of constraints on the
DCGAN is proposed to make this network stable
to train. This architecture is a basic component in
many types of GANs.
■ Auxiliary Classifier GAN (AC-GAN) [108]: Although
the DCGAN can produce convincing image samples, it is an intractable problem that GANs cannot
FOURTH QUARTER 2019

IEEE Circuits and Systems Magazine - Q4 2019

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2019