IEEE Circuits and Systems Magazine - Q4 2020 - 73

[52]. However, the adversarial learning also results in new
problems. For instance, the training of GAN requires G and
D being optimized iteratively, which is time-consuming.
Moreover, if we adopt the objective function in Eq. 15, the
training objective is equivalent to minimizing the JensenShannon (JS) divergence between the real data distribution
pdata and the generator distribution pG. While if we adopt
the objective function in Eq. 16, then the training objective is to minimize the Kullback-Leibler (KL) divergence
between pdata and pG and maximize the JS divergence between the two distributions simultaneously. In the first
case, using JS divergence as the loss function can lead to
gradient vanishing because when the two distributions
have significant difference, the JS divergence is close to a
constant. In the second case, due to that the KL divergence
is asymmetric, the generator prefers to generate samples
with fixed patterns and ignore the sample diversity [54].
C. Evolution of GANs
To fix such theoretical flaws, a mathematical tool is introduced into the framework of GAN. Wasserstein GAN
[55] leverages Wasserstein distance to substitute the KS
and JS divergence as the measurement of the distribution distance. This distance measurement is symmetric
and can reflect the distribution distance effectively even
in case that two distributions gap largely. That is to say,
the Wasserstein distance can fundamentally solve the
problems of gradient vanishing as well as diversity lacking in GAN. Besides, another idea to enhance the model
stability is to raise the resolution of synthesized images
progressively. According to this, a progressive training
mechanism is developed in ProGAN [56], which trains
the generator and discriminator layer by layer.
In image generation, the impressive performance of
GAN attracts a lot of attention and plenty of variants are
presented in recent years. Due to the high flexibility of
adversarial framework, GAN can be combined with CNN
to leverage its ability of image processing and produce
images with better quality [57]. Moreover, variational autoencoder can also be adopted as the generator in GAN
framework, which is less susceptible to the problem of
model collapse [58].
Besides, label information can be introduced into the
input space of GAN to generate samples with specified
type. Conditional GAN [59] decomposes the input space of
GAN into stochastic noise z and class label y. In this case,
the generated image can be modified by manipulating in
the latent space rather than directly in the image space
where the modification is difficult because the image distributions lie in high-dimensional complex manifolds.
In addition, the appearance of GANs leads to better
solutions to a problem called style translation or imageto-image translation. For two different image styles, two
FOURTH QUARTER 2020

GANs are trained in cycleGAN [60]. One GAN translates
the image from the original domain to the objective domain and the other GAN translates the image from the
objective domain to the original domain. Except for the
standard training of GAN, an additional cycle consistency
loss is introduced to reduce the discrepancy between the
input image x and the image generated via bi-directional
translation from x.
In recent years, attention mechanism is widely applied in RNNs, CNNs and other deep neural architectures
to allocate more attention to meaningful information. By
adopting self-attention mechanism, SAGAN [61] learns an
attention map from the feature maps to model the non-local dependencies between feature regions. Thus, SAGAN
can generate details of images considering the information from farther location in the feature maps.
VI. Conclusion and Prospect
With continuously deepening of the research about DNNs,
various deep neural architectures have been presented
and widely applied in many fields. In this paper, we have
reviewed the intrinsic motivation of some widely-used
deep neural architectures. These motivations have promoted the development and innovation of existing neural architectures from different aspects. Using this as a
starting point, we have roughly categorized the influence
domains of motivations for some DNNs as three classes,
namely information processing, information transmission and learning strategy. To further illustrate how neural architectures are motivated and developed from the
three aspects, convolutional neural network, recurrent
neural network and generative adversarial network have
been taken as instances and discussed respectively in detail. According to the discussion, some conclusions have
been summarized as follows.
In the aspect of information processing, deep neural
architectures can be developed by improving the working
mechanism of neurons or adjusting the information directly. On the one hand, the ability of feature capturing can be
enhanced by changing the sample mechanism of neurons.
For example, sampling one-dimensional vector globally is
transformed to sampling two-dimensional plane locally in
CNNs. On the other hand, representations can be modified
by a learned mask based on attention mechanism.
In the aspect of information transmission, network motivations focus on the utilization of features through changing the path structure of information flow in DNNs. Appropriate path connection cannot only improve the network's
ability of representation learning, but also achieve the embedding of special data such as time series and graphs.
In the aspect of learning strategy, according to specific task requirements, the proposal of a novel learning
strategy can be realized by presenting a corresponding
IEEE CIRCUITS AND SYSTEMS MAGAZINE

IEEE Circuits and Systems Magazine - Q4 2020

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2020