IEEE Geoscience and Remote Sensing Magazine - June 2021 - 77

where Us s =1
" , denote the projections linking to the shared
m
features for different modalities. To avoid overfitting of the
model and to stabilize the learning process, P can be regularized
by the Frobenius norm [169] or the
, norm [171],
11 ,
W = () ,,PP or P 11,
2
F
and Us s^h=
X " ,m
1
modal data, which is written as
X^ " ,m
f
UUYLYU h ,
s s =1h= tr^
where UU ,, ,UUm12=6
Y=
T
S
S
S
SS
@ and
RY1
Y2
hh
g
g
j
g
h
V
Ym
X
W
W
W
WW
.
Similar to Figure 7, L is a joint Laplacian matrix.
Using the general model
R R
(34)
(33)
is specified as an MA term on the multi[198],
LeMA [170], [199], and GiAL [172]. Three common
indices, e.g., OA, AA, and l, are adopted to quantify performance
classification using the support vector machine
classifier on the 2013 Houston HS-MS data sets that have
been widely used in many research studies [169]-[172].
Table 5 provides a quantitative comparison among the
in (32), the following have
been proposed.
◗ The work in [169] considers the HS-MS correspondences
that exist in an overlapped region as the model
input. The learned, shared representations (e.g.,
XU )Yss
=
can then be used for classification on a larger
area even though only MS data are available in the
inference phase.
◗ Differently, the work in [170] inputs not only the labeled
HS-MS pairs but also the unlabeled MS data in
large quantities. With graph learning, i.e., the variable
L is to be learned from the data rather than fixed by a
given RBF, the unlabeled information can be made useful
to find a better decision boundary. According to the
equivalent form of (34), we then have
tr^UYLY UW Wd 1,1
R R == 9
h
2
1 tr d 2
1
^
where dx x=- denotes the pairwise distance in
ij, ij 2
2
Euclidean space. Using (35), the resulting optimization
problem with respect to the variable W is
s.t.WW WW
Wd
2
1
9
11,
==$ij,
11,
R ,, .c
(36)
◗ Inspired by the brain-like feedback mechanism presented
in [89], a more intelligent CML model was proposed
[172]. With the joint use of labels and pseudolabels updated
by the graph feedback in each iteration, more representative
features can also be learned (even if a certain
modality is absent, i.e., in the CML case).
EXPERIMENTAL STUDY
We evaluate the performance of several SOTA algorithms related
to the CML issue both quantitatively and qualitatively.
They are O-baseline (i.e., using original image features),
USMA [85], SMA [160], [197], SSMA [163], CoSpace [169],
JUNE 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
TABLE 5. A QUANTITATIVE COMPARISON OF SOTA ALGORITHMS
RELATED TO THE CML'S ISSUE IN TERMS OF OA, AA,
AND l USING THE NN CLASSIFIER ON THE 2013 HOUSTON
DATA SETS. THE BEST ONE IS SHOWN IN BOLD.
METHODS
O-baseline
USMA [85]
SMA [160]
SSMA [163]
CoSpace [169]
LeMA [170]
GiAL [172]
OA (%)
62.12
65.54
68.01
69.29
69.38
73.42
80.66
AA (%)
65.97
68.81
70.5
72
71.69
74.76
81.31
l
0.5889
0.6251
0.652
0.6659
0.6672
0.711
0.7896
h
,
(35)
aforementioned methods for CML-related classification,
while Figure 14 visualizes a region of interest of classification
maps. By and large, the classification accuracy of
O-baseline, i.e., using only MS data, is much lower than
the other methods. By aligning multimodal data on manifolds,
MA-based approaches perform better than O-baseline
with an approximated increase of 3% OA in USMA,
6% OA in SMA, and 7% OA in SSMA. As expected, the
performance classification of SSL-based models, e.g., CoSpace,
LeMA, and GiAL, is obviously superior to that of
the MA-based ones. In particular, GiAL dramatically outperforms
other competitors due to the use of the brainlike
feedback mechanism and graph-driven pseudolabel
learning. Visually, shared learning methods tend to capture
more robust spectral properties and achieve more realistic
classification results. As can be seen from Figure 14,
the shadow region covered by clouds can be finely classified
by CoSpace, LeMA, and GiAL while the MA-based
models fail to identify the materials well in the region.
REMAINING CHALLENGES
CML has drawn continued interest from researchers in
computer vision and ML, yet it is rarely investigated in the
RS community. In other words, CML is a relatively emerging
topic in RS, which means there are many difficulties
(or challenges) to be overcome.
◗ Data preparation: Multimodal data are acquired with different
contexts, sensors, resolutions, and so forth, which
inevitably poses a great challenge to data collection and
processing. For example, the errors caused by the interpolation
of different resolutions, the registration methods
of geographical coordinates, the pixelwise biases of
different sensors, and the uncertainties of image degradation
in the imaging process easily generate unregistered
multimodal data to a great extent.
77
IEEE Geoscience and Remote Sensing Magazine - June 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2021
