IEEE Geoscience and Remote Sensing Magazine - September 2022 - 226

compared to the nonadversarial DCSTFN. Bouabid et al.
[162] propose a model similar to the popular pix2pix GAN
[163], which comprises a conditional GAN with a U-Net architecture
for the generator and a PatchGAN architecture
for the discriminator.
Chen et al. [155] (CycleGAN-STF) employ a cycle GAN
architecture [164] to enhance the traditional flexible spatiotemporal
data fusion (FSDAF) algorithm [165]. The main
framework consists of the following four stages:
1) Generation: A cycle GAN takes as the input the HSLT image
pair (, )FF
tt
-+
11 and produces an Ft
GAN
in the output.
The GAN produces a single image each time, so an iterative
generation scheme is introduced to generate multiple
in-between images.
2) Selection: A single Ft
GAN
outperformed the other approaches in preserving spatial
details but resulted in a loss of spectral information.
Zhang et al. [156] (STFGAN) propose a cascade of two
SRGAN-like structures that learn to produce an HR Landsat
image for a target date t2
based on Landsat-5/7 data from
tt
12
t 2
F .int
dates t1 and t3 as well as MODIS data from dates ,, and
t .3 The first GAN takes as the input the two Landsat and
all of the corresponding MODIS images and produces
an intermediate Landsat image
ability of the SRGAN for spatial enhancement to such a
large scaling factor (),16#
t 2
this image is far from optimal.
image is selected based on mutual
information metrics of the HSLT and LSHT images.
3) Enhancement: The discrete wavelet transform is used to
enhance the quality of the selected image, borrowing
information from
C .t
4) Fusion: The result of the previous steps along with Ct
are inserted in the FSDAF algorithm to obtain the
and
Ct 1final
prediction.
The model was only compared with traditional non-DL
algorithms. Experiments showed that CycleGAN-STF
Therefore, a second GAN is used that takes as the input the
Landsat images along with a downsampled version of these
Landsat images and the intermediate Fint
image.
final F2
A different approach is followed by Tan et al. [166]
(GAN-STFM), who propose a conditional GAN architecture
for downscaling MODIS images with a Landsat reference.
The generator follows a U-Net architecture, and the inputs
are the coarse MODIS image at the prediction date t Ct
a fine Landsat image at a different date t)
or the coarse Ct
to produce the
Due to the limited
and
arbitrarily close
to the target F .t) Similarly, the discriminator takes as the
input a concatenation of either the coarse Ct
sponding ground truth Ft
ed Ft
pred to perform a fake/real clasNonlinear
Mapping
CNN
MODIS
Images
Learning a Nonlinear
Mapping CNN
LSR Landsat
Images
Learning an SR CNN
Landsat
Images
Training Stage
Transitional LSR Landsat
Images on t1, t2, and t3
Fusion Model
LSR Landsat
Image on t2
SR CNN
Transitional Landsat
Images on t1, t2,
and t3
Fusion Model
Predicted Landsat
Image on t2
Prediction Stage
FIGURE 23. An outline of the STFDCNN method. (Source: [154]; used with permission.)
226
Landsat
Images on t1 and t2
LSR Landsat
Images on t1 and t2
MODIS Images
on t1, t2, and t3
sification. All convolutional blocks
in both networks are replaced by custom
residual blocks with switchable
normalization [167] in the generator
and spectral normalization [168] in
the discriminator.
The authors further propose the
use of a multiscale discriminator
where all inputs are additionally
downsampled with factors /2 and /4
and are used to train three different
discriminators with similar architectures
at different scales. The proposed
method is compared with non-DL
approaches and EDCSTFN, showing
the superiority of the random
Landsat reference selection against
the temporal proximity imposed by
STF in terms of computational cost
without compromising the downscaling
quality.
Different DL approaches for
blending Landsat-8 with Formosat-2
(8 m) images to increase the number
of cloud-free observations have been
studied by Teo and Fu [169]. First,
Landsat images were resampled to 8
m and then blended with the rest via
a simple STARFM algorithm. Second,
pairs of Formosat and Landsat images
obtained on the same date were
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022
and the correand
the predict
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
