IEEE Geoscience and Remote Sensing Magazine - June 2022 - 305

advantages on correlation coefficient, spectral angle metric
(SAM), and root-mean-square error (RMSE) indexes.
Some of methods in Table 3 are variants of PAN-sharpening
methods. There are some different characteristics between
HS-MS fusion and PAN sharpening that would also
be challenges for fusion algorithms.
First, HS-MS fusion often faces multitemporal problems
since the HS and MS images of the same scenario
are often acquired at different times. The differences
from multitemporal images may be a very challenging
problem because it leads to obvious errors in fusion,
but there is still no mechanism to deal with it in most
current fusion methods. In particular, when the ground
objects are changed, determining how to use a GAN or
more auxiliary data to decrease errors in fusion is still an
open problem.
Second, HS-MS fusion often faces larger spatial resolution
differences than PAN sharpening. The spatial downsampling
factor between PAN images and MS images is
often four. However, for HS and MS images, the factor is
often much higher than four. Too-large spatial differences
will make the ill-posed problem more serious so that it
will lead to severe spatial distortions in the fusion results.
We believe that more efforts need to be made to solve this
problem. The effectiveness of the traditional spatial degradation
model may be very limited in reducing spatial distortions.
GAN-based methods provide us with a new way
to solve the problem of large spatial differences in HS-MS
image fusion.
SPATIOTEMPORAL IMAGE FUSION
Owing to technical and budget limitations, spatial and
temporal resolution capabilities are mutually restricted.
In many cases, satellites with high-spatial-resolution sensors
require long revisit cycles, which provide low temporal
resolutions, while satellites employing short-time
revisit cycles often provide only low- or middle-spatialresolution
images. Therefore, it is often difficult to acquire
high-spatiotemporal resolution (HSTR) data from a
single satellite.
Spatiotemporal data fusion is a feasible solution for this
problem. This technique provides HSTR data by fusing
high-frequency, low-spatial-resolution images with lowfrequency,
high-spatial-resolution images (Figure 11). The
land-surface reflectance between the Landsat Enhanced
Thematic Mapper Plus (ETM+) and MODIS basically satisfies
the complementarity of the spatial and temporal resolutions.
Many studies took Landsat and MODIS data as examples
of spatiotemporal image fusion (Table 4).
Over the past decades, a variety of spatiotemporal data
fusion methods have been developed. The existing spatiotemporal
data fusion methods can be categorized into different
groups: linear blending methods, such as [83]-[85];
unmixing methods [86]-[88]; Bayesian methods [89], [90];
sparse approximation methods [91], [92]; deep learning
methods [93]; and so on.
JUNE 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
Most of these mentioned methods have made some
progress in spatiotemporal fusion. However, it is still
an open problem since it is hard to accurately establish
the complex relationship between high- and low-resolution
images. Current methods often suffer from one
important limitation [94]: all of them have to make an
assumption that some key variables remain unchanged
in the spatiotemporal images. For example, linear mixture
methods assume that the land cover type does not
TABLE 3. THE QUANTITATIVE RESULTS OF THE SIMULATED
EXPERIMENTS FROM [55].
METHOD
GSA [63]
CC
SAM RMSE
ERGAS
PEAK SIGNALTO-NOISE
RATIO
0.9448
3.9458 0.0532 4.2421 31.4454
MTF-GLP [78] 0.9378 4.1096 0.0528 5.3026 29.4774
GFPCA [79]
0.8885 5.3771 0.0715 7.6942 26.3611
HySure [80] 0.9361 4.7074 0.0608 4.8385 30.0765
CNMF [81]
DDLPS [82]
0.9157 4.0742 0.0617 5.5231 29.054
0.9316 4.151
HPGAN [55] 0.9528 3.4967 0.0477 4.602
0.0492 8.7823 28.7133
30.7059
CC: correlation coefficient; MTF-GLP: MTF generalized Laplacian pyramid; GFPCA: guided
filter PCA; HySure: hyperspectral superresolution; CNMF: the coupled nonnegative matrix
factorization; DDLPS: detailed-based deep Laplacian pansharpening; HPGAN: hyperspectral
pansharpening framework based on a 3D GAN. Source: [55].
(a)
(b)
FIGURE 11. Two kinds of spatiotemporal data fusion: (a) two pairs
and (b) one pair.
TABLE 4. THE PERFORMANCES OF DIFFERENT SPATIOTEMPORAL
FUSION METHODS ON THE COLEAMBALLY IRRIGATION
AREA DATA SET [97].
DATE MODEL
FSDAF [102]
BiaSTF [96]
1 April
EDCSTFN [44]
MAE
RMSE
0.01996 0.0297
SAM
0.02117 0.03032 0.0785
0.01972 0.029
SSIM
0.08182 0.93694
0.94014
0.08232 0.94226
GAN-STFM [100] 0.02332 0.03305 0.10131 0.92563
SSTSTF [103]
0.01975 0.02953 0.08116 0.94186
MLFF-GAN [97] 0.01898 0.02781 0.07647 0.94453
*Since BiaSTF uses two reference time image pairs to predict the results, only the average
value of the first four days is calculated here. FSDAF: flexible spatiotemporal data fusion;
BisSTF: bias-driven spatiotemporal fusion model; EDCSTFN: enhanced deep convolutional
spatiotemporal fusion network; GAN-STFM: GAN-based spatiotemporal fusion model;
SSTSTF: spatial, sensor, and temporal spatiotemporal fusion; MAE: mean absolute error;
MLFF-GAN: multi-level feature fusion with GAN for spatiotemporal remote sensing images.
305
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
