IEEE Geoscience and Remote Sensing Magazine - June 2022 - 299

models have very clear physical meanings, so they have almost
all of the advantages of model-driven methods. However,
for some heterogeneous RS data fusion or RS and GO
data fusion, they cannot be denoted as an explicit model.
Therefore, the effect of the explicit model will be very limited
in some fusion scenarios.
THE IMPLICIT, DATA-DRIVEN APPROACH
For early studies of RS data fusion, the explicit model and
model-driven fusion played more important roles. Either
the observation or data model is the fundamental consideration
in most algorithms or processes of fusion. However,
with the development of deep learning and automatic feature
learning, it is possible to weaken the observation model
H. The process of fusion can be easily realized in the hidden
layers of a CNN or other deep architectures, where the
physical meaning of the observation or data-distribution
model is dramatically weakened. We call this kind of trend
of fusion an implicit, data-driven model, which is denoted as
= gg ^^^ hhh
xy ,
F FF1 kK
(13)
RS Image
Fusion With GANs
Homogeneous
RS Data
Spatial-Spectral
Image Fusion
PAN-Sharpening
Image Fusion
Liu et al.
2020 [54]
Xie et al.
2021 [55]
Ma et al.
2020 [56]
Ozcelik et al.
2020 [57]
Zhou et al.
2020 [58]
Shao et al.
2020 [59]
MS-HS
Image Fusion
Yokoya et al.
2017 [68]
Dian et al.
2021 [69]
Imani and
Ghassemian
2020 [70]
Dong et al.
2021 [72]
. . .
. . .
FIGURE 5. A taxonomy of RS image fusion with GANs. NDVI: normalized difference vegetation index.
JUNE 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
299
. . .
Spatiotemporal
Fusion of NDVI
Spatiotemporal
Image Fusion
Spatiotemporal
Fusion of
Reflectance
Song et al.
2021 [97]
Chen et al.
2020 [98]
Zhang et al.
2020 [45]
Tan et al.
2020 [100]
Shang et al.
2022 [101]
Spatiotemporal
Fusion of
Evapotranspiration
SAR and Optical
Image Fusion
SAR-to-Optical
Image Translation
Enomoto et al.
2018 [107]
Merkle et al.
2018 [108]
Reyes et al.
2019 [109]
Turnes et al.
2020 [110]
Fu et al.
2019 [114]
Wang et al.
2019 [115]
2018 [116]
He and
Yokoya
SAR-Optical
Image
Registration
Haydn et al.
2020 [117]
Ren et al.
2020 [118]
where ()Fk g is the kth hidden layer of an implicit model,
such as an CNN. A GAN for RS data fusion is a typical
method of an implicit, data-driven model. In most
cases, we do not need to consider the observation equation
y xn.
=H) +
One of the advantages is that the fusion
process can avoid some of the problems resulting from the
complexity of H, especially for heterogeneous RS data and
so on. At the same time, the explainability or interpretability
of the fusion becomes so weak that we cannot judge how
and why the fusion produces better performances in some
cases. Another problem is that data-driven fusion usually
needs more training data and more computation sources.
Now, data-driven fusion is becoming more popular, but
there are also some studies in which researchers have
started to rethink the relationship between the data- and
model-driven approaches. There are already studies that try
to combine the two schemes in fusion.
TAXONOMY OF RS IMAGE FUSION
There are different taxonomies for RS image fusion if we
take a different perspective. A very common taxonomy of
Heterogeneous
RS Data
RS and GO Data
Point Cloud
and Optical
Image Fusion
Image Fusion
Lidar and
Optical
Zhang and
Lin
2017 [120]
Zhang et al.
2018 [124]
Zhang et al.
2020 [125]
Milz et al.
2019 [126]
Qin et al.
2018 [128]
Depth Point
Cloud and Optical
(RGB-D) Fusion
Liu et al.
2020 [121]
Milz et al.
2019 [126]
Stereo Vision
Point Cloud
and Optical
Image Fusion
. . .
Infrared-Visible
Image Fusion
Infrared and
MS Image
Fusion
Ma et al.
2020 [130]
Liao et al.
2020 [133]
Babu and
Dubey
2020 [139]
Infrared and
PAN Visible
Image Fusion
Li et al.
2020 [136]
Li et al.
2021 [137]
Li et al.
2019 [138]
Ma et al.
2019 [134]
Li et al.
2020 [135]
. . .
. . .
Satellite Aerosol
Optical Depth
GO Data Fusion
Zhang et al.
2021 [18]
RS and Surface
Soil Moisture
. . .
RS and Social
Media Fusion
RS and Traffic
Data Fusion
RS and Physical
Geography
Data Fusion
RS and Human
Geography
Data Fusion
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
