IEEE Geoscience and Remote Sensing Magazine - June 2022 - 313

the two parts of a GAN, which will promote the fusion of
RS and PG data in future research.
RS AND HG DATA FUSION
HG data include communications, social media, locations,
disease transmission, public sentiment, population distribution,
economic information, and so on. Techniques for
acquiring HG data are totally different from traditional data
collection methods, such as RS. Their relationships are so
complicated that they cannot be represented in a uniform
formula or structure. HG data, especially social sensing
data, have developed very fast in recent years. They enable
all citizens to become part of a large sensor network that
is low cost, more comprehensive, and always broadcasting
situational awareness information.
However, data collected with social sensing are often
massive, heterogeneous, noisy, and unreliable from
some aspects; in addition, they often come in continuous
streams and often lack geospatial reference information.
Together, these issues represent a grand challenge
toward fully leveraging social sensing for RS and HG data
fusion. Meanwhile, deep learning-such as long shortterm
memory networks, CNN, GAN, and so on-architectures
become critical components of fusing social
sensing and RS data to understand the HG problem in a
timely fashion.
There are a very wide range of studies for RS and HG
data fusion. In [145], the authors summarized some applications
of RS-HG data fusion in ecological, air, and disaster
monitoring, among others. They also pointed out that the
fusion of heterogeneous RS and social media data exhibits
a huge potential for different applications, especially for
problems in which a (nearly) real-time response is needed.
Very popular and typical applications include population
distribution, public sentiment, flood inundation, urban
functional zone mapping [146], and so on. In [147], the
close relationship between the number of Twitter users and
brightness of nighttime lights over the contiguous United
States was calculated, and geotagged tweets were then used
to up-sample a stable light image for 2013.
In [148], the authors presented and evaluated a method
for rapidly estimating the flood inundation extent based
on a model that fuses RS, social media, and topographic
data sources. In [149], using the extensive flooding (2013)
of the city of Calgary as a case study, it was shown how to
fuse authoritative RS and volunteered geographical data to
estimate the flood extent and identify affected roads during
a flood disaster. In [150], a multimodel fusion neural
network was proposed for estimating fine-resolution population
estimates from multisource data, taking into account
the local spatial information and global information of
each geographic unit by the fusion of RS and social sensing
data.
Like RS and PG fusion, until now, we have still not
found any published work that used a GAN to perform RS
and HG data fusion. The reason may be similar: because
JUNE 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
the structure and physical properties of HG data are more
complicated, it is hard to model the relationship between
HG and RS data by GANs. However, we still believe GAN
architectures are promising in future studies of HG and
RS fusion.
Some works also take RS and GO data as heterogeneous
data, while, in this survey, we use the concept of
" heterogeneous RS " data (such as SAR and optical images)
but not heterogeneous data. We do not think RS and
GO data are homogeneous, but this is a review of RS image
fusion. Actually, we think the differences between RS
and GO data are more obvious than those between SAR
and optical images. Therefore, the representation of RS
or GO data becomes the first
keyword in this type of fusion.
The models or schemes
of the fusion of RS and GO
data are another very challenging
problem. The information
between RS and GO
data needs to be extracted or
integrated in a suitable way
whether they are complementary,
redundant, or cooperative.
Finally, the method of results validation also
needs to be noticed since it is not easy to carry out
simulating experiments as it is with homogeneous RS
data fusion.
EVALUATION METRICS AND BENCHMARKS
FOR DATA FUSION
METRICS FOR DATA FUSION
Most homogeneous RS data fusion can be evaluated
by simulating experiments. This means that the
ground truth image x is assumed to be known in the
simulation. In this case, the metrics are used to measure
the differences between the ground truth image x
and fused image xt . For most spatiotemporal-spectral
fusions, their results are composed of more than one
band. We assume that there are N pixels in each band
image and that there are M bands in the fused image.
RMSE, SAM, erreur relative globale adimensionnelle
de synthese (ERGAS), SSIM, and so on are widespread
quality indexes:
RMSE=-t
SAM arccosdn " ,t 2
N
1
=
ERGAS=
SSIM (, )xx =
t
^
S
S
2
1
xx
xx
2
1 / mean
M
M j =1
2
x
d
x
x
RMSE
nn vv
nn
^22
1h^
++ ++
++
xx 12
2
tt
tt
cc
cc
hh
2
^ vxx
2
x
^
^
j
j
h
h
n
2
(19)
2h
.
(20)
313
/ hh^ hxx 2
^^ii
·t
(17)
(18)
TECHNIQUES FOR
ACQUIRING HG DATA ARE
TOTALLY DIFFERENT FROM
TRADITIONAL DATA
COLLECTION METHODS,
SUCH AS RS.
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
