Signal Processing - November 2017 - 103

In its simplest form, early fusion involves concatenation
of multimodal features as was implemented by Poria et al.
[34]. Early fusion of multimodal data may not fully exploit
the complementary nature of the modalities involved and
may lead to very large input vectors that may contain redundancies. Typically, dimensionality reduction techniques like
PCA are applied to remove these redundancies in the input
space. Autoencoders, which are nonlinear generalizations of
PCA [58], are popularly used in deep learning to extract a
distributed representation from raw data. This idea has been
extended to learn a multimodal embedded space with the
aim to represent multimodal data within a common feature
space [59], [60].
One of the issues faced in early fusion of multimodal data
is to determine the time-synchronicity between different data
sources. Commonly, these signals are resampled at a common sampling rate. To mitigate this issue, Martínez and Yannakakis [61] proposed several methods (convolution, training,
and pooling fusion) to integrate sequences of discrete events
with continuous signals.

input modalities are significantly uncorrelated, of very different dimensionality and sampling rates, it is much simpler
to implement a late-fusion approach for multimodal learning problems. An alternative approach, intermediate fusion,
offers much more flexibility as to how and when representations learned from multimodal data can be fused.

Intermediate fusion
Neural networks transform raw inputs to higher-level representations by mapping the input through a pipeline of layers. Each
layer typically alternates linear and nonlinear operations that
scale, shift, and skew its input, producing a new representation of the original data. In the multimodal context, when all
of the modalities are transformed into representations, then
it becomes amenable to fuse different representations into a
single hidden layer and then learn a joint multimodal representation. The majority of work in deep multimodal fusion adopts
this intermediate-fusion approach, where a shared representation layer is constructed by merging units with connections
coming into this layer from multiple modality-specific paths.
Figure 2(c) illustrates a simple intermediate fusion model with
three modalities. Representations (features) are learned using
different kinds of layers (e.g., 2-D-convolution, 3-D-convolution, or fully connected), and representations are fused using
a fusion layer, also known as a shared representation layer.
This shared representation layer can be a single shared
layer that fuses multiple channels at some depth or could
be gradually fused, one or more modalities at a time. A naïve
concatenation of features or weights in the shared representation layer may lead to overfitting or the network failing to
learn associations between modalities due to distinct underlying distributions. A simple method of improving performance
of multimodal fusion is to apply some form of dimensionality reduction like PCA [63] or stacked autoencoders [10] after
constructing a shared representation layer (or fusion layer)
via simple concatenation of weights from different modalities. This choice of fusing various representations at different depths is perhaps the most powerful and flexible aspect of
deep multimodal fusion as opposed to other fusion techniques.
The advantage of a flexible fusion scheme can be seen in the

Late fusion
Late- or decision-level fusion refers to the aggregation of
decisions from multiple classifiers, each trained on separate
modalities [see Figure 2(b)]. This fusion architecture is often
favored because errors from multiple classifiers tend to be
uncorrelated and the method is feature independent. Various
rules exist to determine how decisions from different classifiers are c- ombined.
These fusion rules could be max-fusion, averaged-fusion,
Bayes' rule based, or even learned using a metaclassifier.
Decision-level fusion was popular in the early- to mid-2000s,
when ensemble classifiers received widespread interest within the machine-learning community.
There have been several works that employ late- or decision-level fusion for deep multimodal learning [33], [43], [62]
in addition to some works listed in Table 3. Based on the
papers that we have reviewed, we do not find conclusive evidence that late fusion is better than early fusion--the performance is very much problem dependent. Undoubtedly, when

Output

Layer k

Decision Fusion
Model

Layer 3
. . . Model K

Model 1

Model 2

Features

Features . . . Features

Layer 2

Data Fusion

Modality 1

Modality 2 . . . Modality K
(a)

Modality 1

Layer 1

Modality 2 . . . Modality K

Modality 1

(b)

Modality 2

Modality 3

(c)

FIGURE 2. An illustration of various fusion models for multimodal learning. (a) Early or data-level fusion, (b) late or decision-level fusion, and (c) intermediate fusion.

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

103

Table of Contents for the Digital Edition of Signal Processing - November 2017