IEEE Geoscience and Remote Sensing Magazine - June 2021 - 75

HS image. How best to mine the essence of HS images
and develop efficient regularizers for large-scale processing
still remains a challenge.
◗ Evaluation: In real cases, enhanced HrHS pictures from
the HS and MS images do not exist as the reference images
in the real scenario. How best to evaluate the final
enhanced HrHS images is also a key problem for the future
fusion development approach of HS-MS.
CML FOR LARGE-SCALE LAND COVER MAPPING
With the ever-growing availability of diverse RS data
sources from both satellite and airborne sensors, multimodal
data processing and analysis in RS [156], [157] can
provide potential possibilities to break the performance
bottleneck in many high-level applications, e.g., land
cover classification. HS data feature rich spectral information
that enables high-discrimination ability for material
recognition at a more accurate and fine level. It should
be noted, however, that HS image coverage from space is
much narrower compared to MS imaging due to the limitations
of imaging principles and devices. This means that
HS-dominated multimodality learning (MML) fails to
identify the materials on a large geographic coverage and
even global scale [158]. But, fortunately, large-scale MS or
synthetic aperture radar (SAR) images are openly available
from, e.g., Sentinel-1, Sentinel-2, and Landsat-8. This,
therefore, compels us to ponder a problem: Can HS images
acquired from only a limited area improve the land
cover mapping performance using a larger area covered by
the MS or SAR images? This is a typical issue of CML from
an ML's point of view.
Take bimodality as an example. For simplicity, CML
refers to training a model using two modalities, with one
modality absent during the testing phase or vice versa (only
one modality is available for training and bimodality is for
testing) [159]. Such a CML problem that exists widely in a
variety of RS tasks is more applicable to real-world cases.
Figure 13 shows the differences between MML and CML
in terms of the training and testing process. The core idea of
CML is to find a new data space where the information can
be exchanged effectively across different modalities. Accordingly,
we formulate this process in a general way:
m
, "
=1 s =1
XUs s,mmin / XU s.t. ,, (27)
"
2
1
- YX U ,m
2
ss F
s s =1 !C
where m is the number of input modalities. For simplicity,
we consider only the bimodality case in this topic, i.e.,
m .2=
According to the different learning strategies on modalities,
CML can be roughly categorized into two groups: MA and
shared subspace learning (SSL). The differences between the
two types of approaches mainly lie in the following:
◗ MA learns low-dimensional embedding by preserving
the aligned manifold (or graph) structure between different
modalities. During the process of graph construction,
the similarities between samples [unsupervised
JUNE 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
MA (USMA)] and indirect label information [supervised
(SMA) or semisupervised MA (SSMA)] are used. Despite
the competitive performance obtained by MA-based
approaches for the CML task, the discrimination ability
of the learned features remains limited due to a lack
of directly bridging low-dimensional features with label
information.
◗ SSL, as the name suggests, aims to find a latent shared
subspace where the features of different modalities are
linked via an MA regularizer. Also, the learned features
are further connected with label information. The two
MS Image
Model
Ground Truth
HS Image
(a)
MS Image
Model
Ground Truth
HS Image
(b)
MS Image
Model
?
Missing
HS Image
(c)
FIGURE 13. The model's training and testing in MML- and CMLbased
classification tasks (taking the bimodality as an example). They
share the same training process, i.e., the two modalities are used for
model training. The main difference lies in the testing phase: MML
still needs the input of two modalities while one modality is absent
in CML. (a) The training for MML and CML. (b) The testing for
MML. (c) The testing for CML.
75
Ground Truth
IEEE Geoscience and Remote Sensing Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2021
