IEEE Geoscience and Remote Sensing Magazine - September 2022 - 217

resampled to 10 m, again using k-NN interpolation, and are
concatenated with the native 10-m Sentinel-2 bands and the
downscaled 20-m Sentinel-2 bands. These are fed to the ESRCNN,
which outputs a downscaled version of the Landsat
bands 1-7.
A distinct advantage of this method versus traditional
approaches is the ability to fuse Sentinel-2 and Landsat
data obtained on different, albeit close, dates. Using the
same satellite sources, Chen et al. [2] propose the fusion
of Sentinel-2 and Landsat images to enhance the latter to a
spatial resolution of 10 m. They proved that an adversarial
approach is superior to a nonadversarial one, and the proposed
model resembles the architecture of the ESRGAN
trained on a composite of the RGB bands for both satellites.
The authors also tested whether the GAN model could be
improved by pretraining on natural instead of satellite images
using the DIV2K data set (see the " Data Sets " section),
but the results were not favorable.
In their study, Dong et al. [104] (RRSGAN and RRSNet)
argue that RS images coming from different sources must
be carefully aligned before processing due to differences in
the altitude, viewpoint, or angle. They form a data set consisting
of WorldView-2 (0.5-m) and GaoFen-2 (0.8-m) observations
as well as the corresponding images from Google
Earth (0.6 m). The proposed model is a GAN where image
alignment is assisted by the extraction of gradients.
In particular, a CNN is fed the input images and their
gradients and proceeds to extract features that are then
aligned via a pyramid with deformable convolutional
layers [105]. Subsequently, a relevance attention module
is proposed to combine the aligned features by focusing
on the relevant information, and a series of upsampling
blocks performs the final downscaling. For the adversarial
training, two discriminators are employed, one for the
downscaled image and one for the gradient of the downscaled
image produced by the generator. The loss function
is a weighted sum of: 1) the MAE between the downscaled
and HR images, 2) the adversarial loss for the downscaled
and HR images, 3) the VGG loss between the downscaled
and HR images, 4) the MAE between the gradients of the
downscaled and HR images, and 5) the adversarial loss
for the gradients of the downscaled and HR images. The
results show that both the adversarial RRSGAN and the
nonadversarial RRSNet perform better than numerous
other DL methods, with RRSGAN producing more highfrequency
details.
In conclusion, considering single-source data for MS image
fusion, the available solutions cover a variety of needs.
For example, when all LR input images have the same spatial
resolution (e.g., 20 m), then SPRNet seems to be a more
suitable and robust approach. On the other hand, when
hardware and/or time restrictions apply, FUSE provides a
lightweight candidate since it contains very few trainable
parameters (~28,000) compared to other methods but
has only be applied with a 2# scaling factor. Finally, for
an end-to-end approach where all multiresolution input
bands are downscaled in a single forward pass, FusGAN
seems to produce more accurate and sharp results. In the
case of multisource input data, ESRCNN tackles the lack
of clear, cloudless HR input images on the required date
by enabling the use of multiple HR images acquired at arbitrarily
close dates. The authors observe that, especially
when more than three Sentinel-2 images are used, the model
is able to additionally capture land use/land cover changes
Input Data
G-Net
Upsampling
Global Skip Connection
Addition Conv
Output Image
Lower-Resolution
Spectral Bands
Fused
Concatenation Conv ReLU ResDensBlock ResDensBlock
Scaling
× 4
Higher-Resolution
Spectral Bands
FIGURE 14. The FusGAN generator network. ResDensBlock is the RDB as described in ESRGAN. (Source: [98]; used with permission.)
G-Net: generator of the model.
SEPTEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
217
HR Image

IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022