Signal Processing - November 2017 - 98

Table 1. A comparison between deep multimodal learning and conventional approaches.
Deep Multimodal Learning

Conventional Multimodal Learning

Both modality-wise representations (features) and shared (fused)
representations are learned from data.

Features are manually designed and require prior knowledge about the
underlying problem and data.

Requires little or no preprocessing of input data (end-to-end training).

Some techniques, like early fusion, may be sensitive to data preprocessing.

Implicit dimensionality reduction within architecture.

Feature selection and dimensionality reduction are often explicitly performed.

Supports early, late, or intermediate fusion.

Typically performs early or late fusion.

Easily scalable in terms of data size and number of modalities for
all fusion methods.

Early fusion (data-level fusion) can be challenging and not scalable: latefusion rules may need to be defined.

Fusion architecture can be learned during training.

Rigid fusion architecture usually handcrafted.

Deeper, complex networks typically require large amounts of training
data (if trained from scratch).

May not require as much training data.

Numerous hyperparameter tunings vital for state-of-art performance.

May not have as many hyperparameters as deep-learning architectures.

Compute intensive, requires powerful graphics processing units (GPUs)
for reasonable training time.

GPUs may offer speed-up, but not critical.

the question remains: at which depth of representation would
the fusion be optimal?
The second architectural design choice for deep multi-
modal learning concerns which modalities to fuse. The un--
derlying assumption in multimodal fusion is that different
modalities provide complementary information toward solving the task at hand. However, it could be the case that the
inclusion of all available modalities ends up being detrimental to the performance of the machine-learning algorithm-
and, as such, some form of feature selection may be required.
In the section "Fusion Structure Learning and Optimization,"
we discuss a number of techniques that, during training, can
automatically learn the optimal ordering and depth of fusion.
The third design choice involves dealing with missing data or
modalities. Deep multimodal learning models should be robust
enough to compensate for missing data or modalities during
inference. Generative models are typically used in such instances.
Most deep multimodal learning approaches also involve representation learning from raw data. It is often the case that a deep
multimodal architecture utilizes several standard modules or
"building blocks" that are optimized for a specific kind of data.
The choice of which deep-learning module is best for extracting
pertinent information for a given modality is also an important
architectural design choice. For example, when two-dimensional
(2-D) pixel-based data are considered, convolutional architectures are often preferred. Three-dimensional (3-D) convolutional
networks can be used for volumetric data, like CT, MRI, or even
video. When temporal data are used, variants of recurrent neural
networks (RNNs) such as long short-term memory (LSTM) or
gated recurrent units can be incorporated.
The choice of modality-wise deep-learning architecture
is mainly dependent upon the dimensionality of the input or
whether temporal trends need to be learned. Beyond these common architectural choices, it is up to the reader to decide, given
application-specific requirements that may involve properties of
the data set or even the hardware used for training or deployment.
98

Applications
This section aims to provide an overview of the various application domains where deep multimodal learning has garnered
much interest. Although multimodal learning and fusion is a
widely researched topic, deep multimodal learning only started to gain attention following the works of Ngiam et al. [6]
and Srivastava and Salakhutdinov [7]. These early works on
deep multimodal fusion involved only two modalities: images
and text. Ngiam et al. [6] investigated several approaches for
multimodal fusion that include simple concatenation of inputs
and shared representation learning, as well as cross-modality
learning (where data from all modalities are present during
training, but only a single modality is available during test).
At around the same time, Srivastava and Salakhutdinov [7]
also demonstrated the utility of fusing higher-level representations of disparate modalities involving images and text in a
deep-learning framework. A notable finding was that constructing a multimodal fusion layer by way of simple concatenation
of incoming connections resulted in relatively poorer results-
revealing that hidden units have strong connections to variables
from individual modalities but few units that connect across
modalities. They also found that capturing cross-modality correlations required at least one nonlinear stage to be successful
since higher-level representations of individual modalities will
be relatively "modality free" and therefore more amenable to
fusion. These early explorations became the basis of a number
of proceeding works in deep multimodal learning that investigated various regularization strategies (see the section "Multimodal Regularization") to enforce constraints for learning
intermodality relationships.

Human activity recognition
An important area of research that heavily utilizes multimodal
data is human activity recognition. Under this large umbrella
of research, there are numerous subfields of research that relate
to some aspect of human understanding. Given that humans

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017