IEEE Geoscience and Remote Sensing Magazine - September 2022 - 229

observations. The final prediction is obtained by summing
the output of the two networks along with the initial HSLT
image. The results showed that this inclusion of the sensor
bias lets the model converge to a lower minimum, and its
predictions exhibit fewer spatial and spectral distortions.
In conclusion, the studies presented in this section provide
a variety of methods for tackling the spatiotemporal
variation of the observed landscape. The lack of a common
benchmark data set, again, renders the direct comparison
of all methods infeasible, but certain useful characteristics
can be discerned. First, models such as EDCSTFN, GASTFN,
and GAN-STFM require a minimal number of input
images, thus facilitating the downscaling task in areas with
severe cloud contamination. Among these approaches,
GAN-STFM has the additional advantage of using fine images
at arbitrary dates prior to the target date, which provides
an extra level of freedom concerning the selection of
images for training and/or inference.
Second, EDCSTFN, DMNet, STF3DCNN, and BiaSTF
employ simple architectures with a limited number of
trainable parameters, which makes them ideal candidates
for quick experimentation and testing. Finally, considering
the spectral correlation between the different bands enables
the model to exploit complementary information to better
uncover land cover and phenological changes. The models
accepting multiband input are EDCSTFN, GASTFN, STFGAN,
GAN-STFM, DL-SDFM, and STF3DCNN.
SUPER-RESOLUTION
SR is a broad family of methods that aim to enhance the
spatial resolution of an image without the need to blend
information from auxiliary sources in either the spectral
or the temporal dimensions. For better assessment, they
can be categorized into SISR, multiple-image SR (MISR),
and reference SR (RefSR). These are presented next, while,
in Table 5, we summarize the main DL models developed
for SR. In the " Super-resolution for Synthetic Aperture Radar
and Aerial Imagery " section, we examine SR architectures
that are specific for synthetic aperture radar (SAR)
and aerial imagery.
SINGLE-IMAGE SUPER-RESOLUTION
SISR aims to recover an HR version of a single LR input image.
However, lost pixel information in the LR image can
never be fully retrieved but only hallucinated, which means
that multiple possible HR images can be constructed from
one LR source. This renders the SISR problem mathematically
ill posed and noninvertible, but it is often the only
viable approach when only a single LR input is available.
Therefore, several attempts have been made to employ DL
techniques in the SISR domain for RS.
MULTISCALE APPROACHES
Lei et al. [171] (LGCNet) (Figure 26) design a CNN model
that combines feature maps produced by previous layers to
extract information at different scales and levels of detail.
SEPTEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
The model was evaluated on the University of California
(UC), Merced data set and selected Gaofen-2 images, and
it managed to outperform traditional image enhancement
methods, such as bicubic interpolation and sparse coding,
but showed only marginal improvements compared
to other established DL models. Haut et al. [172] experiment
on the same data with a residual model containing a
sequence of convolutional layers for feature extraction and
an inception module followed by upsampling layers for the
final downscaling. Their method achieved a performance
similar to that of LGCNet.
Lu et al. [173] (MRNN) propose a preupsampling architecture
with parallel convolutional layers and design a
network with three parallel branches containing residual
blocks of different convolutional kernel sizes. Each branch
is initially trained separately with interpolated versions of
the original LR image varying in size, and then all branches
are combined for the final image reconstruction and finetuned
in an end-to-end setting. Experimental results show
promising improvements over other state-of-the-art DL
methods, especially for larger scaling factors. In another
multiscale approach, Xu et al. [174] employ a U-Net-resembling
architecture, adding a module with sequential dilated
convolutions at the bottleneck section, a global residual
connection, and pixel shuffle operations before the final
output. The dilated convolutions have different dilation
rates, allowing the model to extract information using different
receptive fields and scales.
MULTITASK LEARNING
In their study, Yan and Chang [175] (MSF) exploit a multitask
learning procedure to improve the generalization of
the underlying network to different degradation models.
According to the standard approach, an image is downsampled
by convolving with a Gaussian blur kernel, applying
bicubic interpolation and then adding some noise. The
authors argue that a model trained on images degraded by
a single Gaussian kernel may perform quite well on such
images but fail to generalize to different kernels. Therefore,
they propose a model trained in a multitask setting
where each task represents a separate Gaussian kernel and
is learned by a dedicated CNN.
ADDITIONAL POSTPROCESSING
A study by Qin et al. [176] (DGANet-ISE) presented a custom
postprocessing pipeline for the improvement of the
output of an SR model. Their architecture is heavily based
on EDSR (see the " Standard Deep Learning Methods for
Downscaling in Computer Vision " section) and is trained
with a custom loss function that additionally considers the
gradient similarity between the prediction and target. The
model's output is then iteratively improved via a proposed
image-specific enhancement (ISE) algorithm that backprojects
the error between the SR output and the LR input
image and, accordingly, updates the prediction. This algorithm
alleviates the possible variation between the training
229
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
