Signal Processing - November 2017 - 102

backpropagation algorithm or by using the learned representation in conjunction with other classifiers such as support vector
machines (SVMs).
For multimodal learning problems, generative models are
useful in situations where there could be missing modalities
during test time or when there is a lack of labeled data. The
early works of Ngiam et al. [6] and Srivastava and Salakhutdinov [30] proved that generative models are indeed capable
of handling such learning problems. Since then, a number of
works have been reported in the literature that specifically
deal with using generative deep multimodal networks in
cases where there are missing data [31], [45].
While energy-based models based on stacking RBMs have
received most of the attention in deep generative multimodal
learning, the landscape of generative models is changing. Re--
cently, generative adversarial networks [46], deep directed
models trained with variational inference [47], are gaining
traction in multi- and unimodal settings [48]-[50].

Hybrid models
While discriminative models are trained to maximize the separation between classes, generative models excel at modeling
data distributions. Hybrid models combine both discriminative and generative components in a unified framework. Deng
[51] defines hybrid deep architectures as architectures where
the goal is discrimination but is assisted (often in a significant
way) with the outcomes of generative architectures. For example, the generative component in a hybrid model may learn a
deep representation of input modalities and use the discriminative component for classification or regression tasks.
Hybrid models can be divided into three groups as per [52]:
1) joint methods that optimize a single objective function to
learn a joint representation using the generative and discriminative components
2) iterative methods that learn a shared representation using
an iterative method such as expectation maximization
using representations updated from both generative and
discriminative components
3) staged methods, where the generative and discriminative
components are trained separately in stages.
Representations learned by the generative model in an unsupervised manner can then be used as features for the discriminative component using supervised training.
An example of a joint model is reported in [53], where
short-term temporal characteristics and long-range temporal
dependencies for audio-video modalities are modeled by combining a conditional RBM temporal generative model for the
former and a discriminative component consisting of a conditional random field for the latter. This model also is able to
handle missing modalities due to the generative component.
Other related hybrid architectures include those of Sachan
et al. [54] and Liu et al. [55].

Summary
In this section, we have highlighted multimodal architectures
according to their primary learning paradigm. In some sense,
102

deep-learning models can be thought of as building blocks
that allow us to "mix and match" different models to create
elaborate deep multimodal architectures. While this can be
seen as an advantage of deep learning, a common issue is that
architecture design has been more an art than a science. Not
withstanding, there are numerous hyperparameters associated
with each model that have to be carefully fine-tuned, and this
process may be possibly even more complicated when dealing
with hybrid architectures. Another aspect to be concerned with
is the choice of the fusion structure between modalities and
their representations. Next, we discuss several choices for multimodal fusion structure and direct our discussion to the attractive notion of optimizing and learning this fusion architecture
for improved performance.

Fusion structure
Deep architectures offer the flexibility of implementing multimodal fusion either as early, intermediate, or late fusion. Multimodal fusion approaches predating the advent of deep learning
often referred to early fusion as feature-level fusion and late
fusion as decision-level fusion. With deep-learning approaches, however, the idea of feature-level fusion can be extended
further to the concept of intermediate fusion.

Early fusion
Early fusion involves the integration of multiple sources of
data, at times very disparate, into a single feature vector, before
being used as input to a machine-learning algorithm as illustrated in Figure 2(a). The data to be fused are the raw or preprocessed data from the sensor; hence, the terms data fusion
or multisensor fusion are often used.
If data fusion is performed without feature extraction, this
could be quite challenging. For instance, the sampling rate
between different sensors could vary, or synchronized data
from multiple data sources might not be available if one
source produces discrete data, while another source provides
a continuous data stream.
To alleviate some of the issues related to fusing raw data,
higher-level representations can first be extracted from each
modality, which could be either handcrafted features or learned
representations, as is common in deep learning, before fusing
at the feature level. When nonhierarchical features are used, as
often the case in handcrafted features, features extracted from
multiple modalities can be fused at only one level, prior to being
input to the machine-learning algorithm. Since deep learning
essentially involves learning hierarchical representations from
raw data, this gives rise to intermediate-level fusion.
Most early-fusion models make the simplifying assumption
that there is conditional independence between the states of
various sources of information. This may not be true in practice, as multiple modalities tend to be highly correlated (for
example, video and depth cues). Sebe [56] argues that different streams contain information that is correlated to another
stream only at a high level. An excellent example of this can be
seen in [57]. This assumption allows the output of each modality to be processed independently of the others.

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017