IEEE Geoscience and Remote Sensing Magazine - September 2022 - 234

Finally, based on the popular EDSR architecture, Peng
et al. [198] (PGCNN) propose a gated residual block that
encourages the model to focus on high-frequency details,
whereas Lei and Shi [199] (HSENet) employ custom attention
modules that aim to discover information recurring at
multiple scales inside the image. All of the aforementioned
studies show that the inclusion of such attention mechanisms
boosts the model's performance and helps achieve
a sharper downscaled result closer to the HR ground truth.
RECURSION
Chang and Luo [200] (BCLSR) present a novel approach
by employing a recursive framework on images obtained
from the GaoFen-2 satellite. Their model comprises multiple
densely connected convolutional blocks that share their parameters
and feed their outputs to a bidirectional convolutional
long short-term Memory layer (BiConvLSTM). The
output is then downscaled via a subpixel convolution. The
results show that this method outperformed several established
DL models and produced sharper results without losing
substantial high-frequency details.
GENERATIVE NETWORKS
A multitude of studies have also explored the adaptation
of GAN models for SR. In an interesting approach, Lei
et al. [201] (CDGAN) present the " discrimination-ambiguity "
problem, which states that RS images contain more lowfrequency
components than natural images, thus impairing
the discriminator's ability to decide whether a given input
is real or fake. To tackle this issue, they propose a " coupled
discriminator " that takes as the input both the predicted SR
image and its corresponding HR ground truth shuffled by a
random gate and is then tasked with deciding whether the
input constitutes a real-fake pair (one) or a fake-real pair
(zero). The generator architecture is based on ESRGAN. The
model competed against a number of DL methods on the
UC Merced and Wuhan University-Remote Sensing (WHURS19)
data sets (see the " Data Sets " section) as well as selected
GaoFen-2 images and produced less blurry results with
fewer artifacts.
A number of studies have also proposed minor adjustments
of popular SR architectures to fit the needs of the RS
domain. For example, Ma et al. [202] (DRGAN) utilize an
RDN-like architecture for the generator with subpixel convolution
for downscaling and a VGG loss function. Their
model was evaluated on the NWPU-RESISC45 data set (see
the " Data Sets " section) and several other CV benchmarks
and achieved sharper images with cleaner object boundaries
as compared with other state-of-the-art DL methods.
Salgueiro Romero et al. [203] (RS-ESRGAN) adapt the ESRGAN
model in a preupsampling framework and train the
generator in three stages: first, it is trained on a set of WorldView
images only; then, it is fine-tuned on pairs of WorldView
and Sentinel-2 images; and, finally, it is trained in an
adversarial manner with WorldView and Sentinel-2 pairs.
The final image is formed by a linear combination of the
234
generator's output trained with and without the adversarial
scheme, which helps the user calibrate the perception-distortion
tradeoff.
MULTISCALE GENERATORS
Dense and multilevel connections have also been introduced
to different generator architectures with the aim of
extracting more accurate representations of both small- and
large-scale objects. For example, Wang et al. [204] (udGAN)
design a novel ultradense residual block that contains parallel
convolutions and additional diagonal connections,
while features at each level are concatenated through a bottleneck
1 # 1 convolution to limit the channel size. Their
study illustrates the value of this new connectivity scheme
by surpassing several other established DL methods in the
sharpness and quality of the produced images.
Shin et al. [205] propose a multiscale generator comprising
multiple parallel streams in a pyramidal fashion, each
of which is formed by a series of RDBs. A reconstruction
module fuses the output of all streams and produces the
final SR image. Before entering the discriminator, an HR
or SR image is first fed to a pretrained VGG network, and a
number of intermediate feature maps are selected. A set of
blurring Gaussian kernels is applied on these feature maps,
and the results are then fed to a discriminator model with
a PatchGAN architecture. Both networks are illustrated in
Figure 27. The proposed method achieved much better results
compared to EEGAN and CDGAN, and it managed
to capture and recover even small-scale details in the produced
images, which the other techniques failed to do.
Another multiscale approach was introduced by [206]
(Enlighten-GAN) that improves on the ESRGAN by adding
an " enlighten block " to the generator. This block outputs an
intermediate SR image and helps the generator learn highfrequency
information in a progressive manner. The loss
function has a self-supervised hierarchical perceptual loss
component, where an autoencoder is trained from scratch
on RS images, and the distance between the corresponding
feature maps of the SR and HR images is computed.
Finally, the authors present a novel large image tiling and
batching approach for downscaling overlapping satellite
image patches separately (Figure 28). Experimental results
showed that Enlighten-GAN produces sharper images
with much fewer artifacts than other GAN-based methods
while, at the same time, retaining the true hues and shapes
of the objects.
GENERATIVE ADVERSARIAL
NETWORKS AND ATTENTION
Attempting to improve the output of an SR GAN model,
multiple studies exploit attention mechanisms. Jiang et al.
[207] (EEGAN) propose a generator that first enhances the
input and then extracts and sharpens its edges (Figure 29).
A mask branch with an attention mechanism is also employed
during the edge-enhancement step to focus on
the useful information. The model outperforms SRGAN,
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
