Signal Processing - November 2017 - 104

work of Karpathy et al. [64], who showed that using a "slowfusion" model, where learned representations of video streams
are gradually fused across multiple fusion layers during training, consistently produced better results for a large-scale
video classification problem, as opposed to early-fusion and
late-fusion models. Similarly, Neverova et al. [8] empirically
showed that implementing a gradual fusion strategy, by first
fusing highly correlated modalities, to less correlated ones in
a progressive manner (e.g., visual modalities first, then motion
capture, then audio), produced state-of-art results for communicative gesture recognition.
Although learning a multimodal representation using the
shared representation layer is indeed flexible, many current
architectures require careful design in terms of how, when,
and which modalities can be fused. In the "Fusion Structure Learning and Optimization" section, we discuss further
attempts at optimizing the tedious architecture design process required by multimodal learning.

Multimodal regularization
Deep-learning techniques iteratively optimize a set of model
parameters (typically, the weights and biases between each
layer) by minimizing a loss function. To improve generalization, one or more regularization strategies are employed, often
as an additional term added to the loss function. From a computational perspective, regularization provides stability to the optimization problem leading to algorithmic speed-ups, and, from a
statistical point of view, regularization reduces -overfitting [65].
In the deep multimodal learning context, an important
design consideration is the formulation of cost functions and
regularizers that enforce intermodality and intramodality
relationships such as information-theoretic regularizers and
structured regularization, which we now briefly review.
Information-theoretic regularizers are formulated using
measures such as mutual information and variation of information. For example, Sohn et al. [66] proposed a cost function that minimized the variation of information between
modalities to learn relationships between modalities. The
intuition behind this formulation is that learning to maximize
the amount of information that one data modality has about
the others would allow generative models to reason about the
missing data modality given partial observations. Alternatively, a mutual information term could also be maximized during
training [67]. Another information theoretic loss formulation
based on the Kulback-Leibler (KL) divergence was proposed
by Zhu et al. [68] for a multilabel image annotation problem.
In a pretraining stage, they first trained CNNs to learn intermediate representations from each modality using unlabeled
data and then, in the fine-tuning stage, used backpropagation
to minimize the KL-divergence between the predictive and
ground-truth distributions. Finally, to learn the optimal combination of multimodal weights, they adopted the exponentiated online learning algorithm to sequentially find an optimal
set of combinational weights.
Taking inspiration from structured feature selection in
multitask learning [69], Wu et al. [70] designed a model that
104

uses a trace norm regularization term, which encourages similar modalities to share similar representations for a video
classification problem using video and audio modalities.
Cost functions that enforce inter- and intramodality correlations have also been explored by Wang et al. [71]. Their
formulation includes a discriminative term, and a correlative
term based on canonical correlation analysis. In a follow-up
work [72], they proposed a multimodal fusion layer that uses
matrix transformations to explicitly enforce a common part to
be shared by features of different modalities while retaining
modality-specific learning.
Lenz et al. [28] formulated a structured regularization
term in the cost function, which allows a model to learn correlated features between multiple input modalities but regularizes the number of modalities used per feature, thereby
discouraging the model from learning weak correlations
between modalities. Structured regularization essentially
applies some form of regularization separately for each set of
modality-specific weights. They considered several variants
of structure regularization for a multimodal robotic grasping
task. One that worked well in their case incorporated the L 0
norm on top of the max-norm penalty
f (W) =

S r, i
/ / I $`max
i

j=1 r=1

W i, j j > 0 .,

where S r, i is one if feature i belongs to group r and is otherwise zero. S is a binary modality matrix of size R # N, where
each element S r, i indicates the membership of a visible unit,
x i, in a particular modality, r. I is an indicator function, which
takes a value of one if its argument is true and is otherwise zero.
In some problems, temporal context can play an important
role, for example, driver activity anticipation. Unlike human
activity recognition, where complete temporal context is
available, in driver activity anticipation, the machine-learning system must predict using only partial context within a
short span of time before the event occurs. To solve this problem, Jain et al. [35] incorporated a temporal term that grows
exponentially in time into their cost function for a multimodal RNN with LSTM units. This encourages the model to fix
mistakes as early as it can.
Multimodal-aware regularizers have resulted in marginal to notable improvements in model performance. Despite
including these multimodal regularization strategies, the
deep-learning architectures discussed in this section have
input modalities merging into a single fusion layer. A possible
extension could be to investigate a gradual fusion model that
takes advantage of these regularization strategies.

Fusion structure learning and optimization
Most multimodal deep-learning architectures proposed to
date are meticulously handcrafted. While many models
adopt a single fusion layer (shared representation layer), several stand-out works [8], [64] implemented a gradual fusion
strategy. The choice of which modality is fused, and at which

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017