IEEE Geoscience and Remote Sensing Magazine - September 2022 - 209

POSTUPSAMPLING FRAMEWORK
Mitigating the complexity and high cost of the preupsampling
approach, in the postupsampling framework, an endto-end
model undertakes the upsampling task via trainable
layers located at the end of the architecture. In the most common
approach, a DL network performs feature extraction
on the low-dimensional space of the LR image and finally
increases the resolution to obtain the HR output. A disadvantage
of this framework is the fixed scaling factor, which
forms an integral part of the architecture; thus, a different
model must be designed and trained for different scales. In
addition, performance is highly affected by the magnitude
of the scaling factor. Since upsampling is performed in a
single step, high factors (e.g., ,)
## increase the learning
810
difficulty and make the models considerably harder to train.
PROGRESSIVE UPSAMPLING FRAMEWORK
In this framework, a model upsamples the image in a progressive
manner through consecutive convolutional and
upsampling layers. At each stage, the input is upsampled
to a higher resolution, finally obtaining the required scale
at the output. This approach facilitates the learning process
since the downscaling task is decomposed into much
simpler steps. Such architectures are also able to handle requirements
for multiscale output since each stage produces
an upsampled image of intermediate scale. However, progressive
upsampling models require more complex architectures
and are, thus, harder to design and train.
ITERATIVE UP- AND DOWNSAMPLING FRAMEWORK
This framework exploits consecutive up- and downsampling
layers, which refine the reconstruction error on HRto-LR
projections, thus extracting more information on the
relationship and correlations between the two spaces. Such
models usually achieve higher-quality results and are able
to handle higher scaling factors successfully.
MODELS
One of the first robust DL methods for downscaling was
presented in [73] (SRCNN), where a two-layer CNN was fed
an upsampled version of an image and produced a sharpened
HR output. It was trained and tested on subsets of
ImageNet and outperformed equivalent non-DL methods.
A similar approach was adopted by Kim et al. [74] (VDSR)
who designed a deeper, VGG-like architecture [75] with a
global residual connection and managed to outperform
SRCNN on the test set.
Shi et al. [61], [62] (ESPCN) subsequently introduced
the subpixel convolution, which later became a popular
upsampling technique for DL models. This trick helps reduce
the model's number of parameters without compromising
its representational power.
The next landmark article [76] (LapSRN) introduced a
multiscale architecture that integrates the Laplacian pyramid
structure and produces intermediate images downscaled
by smaller factors (, ,)24 and8## # in a single pass. The
intermediate outputs are supervised via separate Charbonnier
loss functions, and this progressive upsampling scheme
helps the model retain high accuracy in higher scales.
Ledig et al. [77] (SRGAN) introduced an adversarial approach
to spatially enhance natural images. The generator,
named SRResNet, consists of a series of residual blocks,
local and global residual connections, and subpixel convolutional
layers for downscaling. The discriminator is a
VGG-like network that performs the real/fake binary classification.
The generator's loss function is a combination
of the adversarial loss and a term comparing the produced
downscaled and the target HR image. Based on this model,
Wang et al. [78] (ESRGAN) propose a number of improvements
to achieve sharper results. They replace the residual
blocks with novel residual-in-residual dense blocks, which
actually comprise dense blocks with global residual connections,
as seen in Figure 6, and use the relativistic average
discriminator introduced in [79].
Following the success of the baseline SRGAN, Lim et al.
[81] (EDSR/MDSR) extend the SRResNet architecture by removing
the rectified linear unit (ReLU) activations outside
the residual blocks and deepening the model. The authors
name this architecture EDSR and train it separately for the
scaling factors ,,
23
## and #4 . They also noted that, by
fine-tuning a pretrained 2# model when training for 3#
or 4# downscaling, the entire training process is accelerated,
and the algorithm converges much faster. Based on
this observation, the authors argue that downscaling at
multiple scales involves interrelated tasks, so they design an
alternative model, namely, MDSR, which handles multiple
Dense
Block
Dense
Block
Dense
Block
×β
FIGURE 6. The residual-in-residual block (RIRB). It contains multiple dense blocks and residual connections both between blocks and
between the input and output of the RIRB. Here, b refers to the residual scaling parameter. (Source [80]; used with permission.) Conv:
convolutional layer; LReLU: leaky rectified linear unit.
SEPTEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
209
×β
×β
×β
Conv
LReLU
Conv
LReLU
Conv
LReLU
Conv
LReLU
Conv

IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022