IEEE Signal Processing Magazine - January 2018 - 102

as the mechanisms used to generate data,
its ancestors until its descendants. Second,
The major difficulty of
including the uncertainties in the data and
there is no intractable partition function that
learning and inference
in the data generation process. CNNs are
has plagued the undirected models. Last,
with deep directed models but most importantly, latent nodes in the
therefore not directly suitable for generawith many latent variables deep directed generative models are depentive modeling. The latest development in
is the intractable inference dent on each other given inputs (because of
generative adversarial networks (GANs)
[8] can perform effective data generation.
the "explaining away" principle) while it is
due to the dependencies
Based on combining a discriminative CNN
among the latent variables not the case for DBN and DBM. This charwith a generative deconvolutional neuacteristic of the deep directed generative
and the exponential
ral network, GANs produce remarkable
models endows them a powerful ability in
number of latent variable
performance in generating realistic data.
modeling the complex pattern in data.
configurations.
Since GANs employ a simple standardized
Learning and inference in deep directed
random vector to model data uncertainty;
generative models is challenging, mainly
they remain mostly deterministic and cannot fully model the
due to the intractable computation of the posterior probability
uncertainties in data.
of the latent variables and the exponential number of latent conIn this article, we focus on fully probabilistic deep directed
figurations (for binary latent variables). Various approximations
generative models for deep learning under uncertainty that
such as variational inference algorithms [2], [8], [10], [15], [21],
can simultaneously perform data generation and classification
[26] have been proposed to address these challenges. They all
tasks. Based on probability theories, deep probabilistic genuse an auxiliary feed-forward network to approximately solve
erative models offer a probabilistically grounded framework
the intractable posterior probability inference problem. These
to represent, learn, and predict under uncertainty. There exapproximations typically assume the joint latent variable disist several types of probabilistic deep generative models, most
tribution given data can be factorized. The factorized distrinotably, deep Boltzmann machines (DBMs) [29] [Figure 1(c)]
bution, while simplifying learning and inference, sacrifices
and deep belief networks (DBNs) [11] [Figure 1(b)]. While genthe dependencies among the latent variables for efficiency.
erative in nature, these models, for the sake of simplicity in
They inevitably enlarge the distance to the true posterior and
inference, typically assume latent variables are independently
weaken the representation power of the model. This negates a
given data. Such an assumption weakens their data modeling
major advantage of the directed graphical models. Moreover,
and representation power. In contrast, deep directed generative
the existing methods avoid the exponential number of latent
models consist of layers of BNs as shown in [Figure 1(a)]. Comconfigurations, as they mostly deal with real latent nodes.
pared to DBMs and DBNs, the deep directed generative model
Alternatively, the posterior inference can be solved directly,
enjoys several advantages due to its unique way of capturing
without resorting to any other network. Furthermore, efficient
dependencies. First, it specifically models the data generation
learning and inference methods can be designed to maximally
process and allows straightforward ancestry sampling, where
preserve the dependencies among the latent variables. Specifia node is sampled following a topological order, starting from
cally, for learning, data marginal log-likelihood can be maximized directly through a max-max operation to overcome the
exponential number of latent configurations. For inference,
posterior probability inference can be performed through the
pseudo-likelihood, which also preserves dependencies of laH3
H3
H3
tent variables. Furthermore, improved maximum a posteriori
(MAP) inference can be achieved by combining the coordi3
3
3
w
w
w
nate ascent (CA) method with the variational method.
H2
w2

H2
w2

H1
w1

w2
H1

w1
X

(a)

H1
w1

(b)

(c)

FIGURE 1. Graph representation of (a) DRBNs, (b) DBNs, and (c) DBMs,
where arrows represent directed links and line segments represent undirected links. Directed links capture the causal or one-way dependency
between the connected nodes, while the undirected links capture the
mutual dependencies among the connected nodes.

102

Related work
In general, there are three types of deep probabilistic generative models: fully directed, hybrid, and fully undirected. The
DBN [Figure 1(b)] has a hybrid structure, where the top two
layers are connected with undirected links and all other layers are connected with directed links. The DBM [Figure 1(c)]
has a fully undirected structure such that every two consecutive layers are connected using undirected links. In this article,
we focus on the fully directed deep generative models due to
their unique representation power. Depending on the structure
and the type of variables, the directed generative models have
several variants. The most commonly used structure consists
of multiple layers [Figure 1(a)], in which only the bottom layer
represents the visible variables. The connections are directed

IEEE SIGNAL PROCESSING MAGAZINE

January 2018

Table of Contents for the Digital Edition of IEEE Signal Processing Magazine - January 2018