IEEE Circuits and Systems Magazine - Q4 2019 - 29

is that the latent space for generation may not be continuous and easily interpolated. This makes it difficult
to build a generative model and produce variations in
the input image, caused by a discontinuous region from
the latent space.
Compared to the standard autoencoders, VAEs are
able to generate new outputs which are similar to the
training data in a desired direction. The latent spaces of
VAEs are continuous to allow random sampling and interpolation by design. To achieve this unique property,
the output of the encoder (i.e. the latent variables) is
transformed to two vectors as shown in Fig. 11: one for
the means μ and another for the standard deviations v.
These statistical parameters form a vector of random
variables X, of which the probability distribution of all
elements X i is normally distributed. The length of this
random variable vector is the same as the encoding vector in standard autoencoders. The ith element of μ and
v are the mean and standard deviation of the ith random variable X i . After sampling, a real-value vector is
obtained and passed forward to the decoder for generation. Intuitively, the latent representation of an input is
controlled by the mean and standard deviation vector.
Consequently, new samples can be created by the decoder with a slightly varied latent encoding.
To achieve ideal latent variables, which are as close
as possible to each other while still being distinct, the
Kullback-Leibler (KL) divergence is introduced into the
loss function. KL divergence is used to measure by how
much two probability distributions diverge from each
other. A lower KL divergence means the probability
distribution of the generated samples are closer to the
target distribution. For VAEs, the KL loss in a Gaussian
case, as given in (4),
D KL ^Q < P h = 1 / ^v 2i + n 2i - log ^v 2i h - 1 h
2 i=1
n

of simple restricted Boltzmann machines (RBMs). The
edges between layers of a DBN are both directed and undirected. Hidden units are connected to those of other
layers but not with units belonging to the same layers.
To understand DBNs, the definition of RBMs is introduced first. As their name implies, RBMs are derived
from Boltzmann machines, which are stochastic recurrent neural networks. The difference is that in an RBM all
the neurons are paired from two groups of units: visible
and hidden units. In addition, there are no connections
between neurons within a group. This is for a more efficient training algorithm, in particular the gradient-based
algorithm. Fig. 12 shows an example of an RBM. It is
noted that the connections between the visible layer and
the hidden layer are bidirectional. The contrastive divergence (CD) algorithm is most often used to train an RBM
by optimizing the weight matrix [87]. After training, the
hidden layer can perform an exact feature representation of the visible layer and reconstruct the visible layer.
Returning back to the DBN, which is a combination
of multiple RBMs, this composition leads to a fast, layerby-layer unsupervised training procedure. The network
architecture of a DBN is shown in Fig. 13. When the first
RBM is trained by the CD algorithm, another RBM is
stacked, taking the outputs from the hidden layer of the

Input Layer

Encoder

(4)

is the sum of all the KL divergences between the approximate posterior Q and the standard normal distribution P [40]. Obviously, the minima will be reached
when n i = 0 and v i = 1. Besides the KL loss, the reconstruction loss (such as the mean square error) is
also adopted to maintain the similarity of nearby latent variables.
A VAE can also be extended for supervised learning,
namely as the Conditional VAE (CVAE). The CVAE models latent variables and data, both conditioned on some
random variables, which allows producing data with
specific attributes. More details can be found in [86].

μµi

σii

Decoder

Output Layer

2) Deep Belief Network
The deep belief network (DBN) is a type of generative
graphical model and is usually considered as a stack
FOURTH QUARTER 2019

Figure 11. An example VAE configuration.

IEEE CIRCUITS AND SYSTEMS MAGAZINE

IEEE Circuits and Systems Magazine - Q4 2019

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q4 2019