Signal Processing - November 2017 - 83

learning methods such as canonical correlation analysis and
partial least squares, which learn a single linear latent space to
reduce the modality gap, their DCML designs two neural networks to learn two sets of hierarchical nonlinear transformations
(one set for each modality) to nonlinearly map data samples into
a shared feature subspace, under which the intraclass variation
is minimized and the interclass variation is maximized, and the
difference of each sample pair captured from two modalities of
the same class is minimized, respectively. Experimental results
on three different cross-modal matching applications including
text-image matching, tag-image retrieval, and heterogeneous
face recognition demonstrated the effectiveness of the proposed
method. Lin et al. [15], Workman et al. [16], and Vo and Hays [17]
employed DML techniques to address the cross-view matching
problem for image-based geolocalization, in which these methods were used to localize a ground-level query image by matching to a reference database of aerial/overhead images.

Image set classification
Lu et al. [14] presented an MMDML method to recognize objects
form different viewpoints or under different illuminations. Specifically, MMDML jointly learns multiple nonlinear feedforward neural networks, one for each object class, to explicitly
project the instances from each image set into a common feature
space at the top layer of all networks, where the maximal manifold margin constraint is enforced. In this way, class-specific
discriminative information can be effectively exploited for classification. The authors' method achieved competitive performance
on five widely used image set data sets. Figure 8 shows the key
idea of their MMDML method.

applications, visual data are usually captured in wild conditions so that many noisy and low-quality samples are usually
collected, so that it is desirable to develop robust DML methods that can well measure the similarity of these noisy and
low-quality samples. Hence, how to develop robust DML
methods is another interesting future direction for research.
3) Most existing DML methods are developed for a single specific task, which means that a large amount of labeled data
for this task are usually required to exploit the supervision
information. In some real applications, it is difficult to collect
extensive labeled data for a specific task. Therefore, it is
desirable to conduct multitask DML which can leverage
labeled samples from multiple different yet related tasks so
that it is much easier to obtain more labeled samples for
DML, which is also an interesting future research direction.
4) Most existing DML methods are supervised. In many real
applications, it is easier to collect an extensive unlabeled data
rather than labeled data for practical applications. Hence,
how to develop more effective unsupervised or semisupervised DML is an important future direction.
5) Most existing DML methods utilize the contrastive and triplet
loss functions to train deep models. To complete the family of
DML, employing other loss functions (e.g., quadruplet loss
[37]) is also a promising path to the development of DML.

Acknowledgments
We would like to thank the anonymous reviewers for their
insightful suggestions for improving the article. This work was
supported in part by the National Key Research and -Development

Summary and future research directions
In this article, we have summarized the recent trends of DML
and shown their wide applications of various visual understanding tasks including face recognition, image classification, visual search, person reidentification, visual tracking, cross-modal
matching, and image set classification. Empirical results have
clearly demonstrated that DML can significantly improve the state
of the art in these visual understanding tasks.
There are five interesting directions of DML for future research:
1) Most existing DML methods learn one neural network from
a single feature representation and cannot deal with multiple
feature representations directly. In many visual understanding applications, it is easy to extract m
- ultiple features for
each sample for multiple feature fusion. However, these features extracted from the same sample are usually highly
correlated to each other even if they could characterize samples from different aspects. For multiple feature fusion, this
highly correlated information should be preserved because
it usually reflects the intrinsic information of samples. How
to perform DML with multiview feature representation to
preserve the correlation of different features and further
improve the performance is a desirable future work.
2) Most existing DML methods assume that high-quality and
clean samples are usually obtained so that the learned metrics
are employed for visual understanding. In many real-world

Maximal Manifold Margin

FIGURE 8. The basic idea of MMDML for image set classification [14]. MMDML
models each image set as a nonlinear manifold and employs a feed-forward
neural network to nonlinearly map it into a feature space. Assume there are C
classes, MMDML designs C feed-forward neural networks (one for each manifold). At the top layer of the network, the manifold margin is maximized so
that the parameters of these manifolds can be updated with backpropagation.
Finally, the testing image set is fed to each network and the smallest distance
between it and the training class is used for classification.

IEEE SIGNAL PROCESSING MAGAZINE

November 2017

Table of Contents for the Digital Edition of Signal Processing - November 2017