IEEE Geoscience and Remote Sensing Magazine - September 2022 - 208

which operates simultaneously at different scales and exploits
the image difference (residuals) between levels. Applied
to a DL setting, an input LR image is progressively
upsampled s times through convolutional and upsampling
layers, and the residual of each consecutive pair of upsampled
outputs is computed. This results in the production of
s residual images at different scales that contain features at
different levels of abstraction. Such structures have been
extensively used in image downscaling since they split the
problem into smaller manageable tasks of smaller scale and
help the model converge to better optima.
ATTENTION MECHANISM
Through the attention mechanism, the underlying neural
network manages to isolate and focus on the most important
feature details for the task at hand. Multiple types of
attention mechanisms have been proposed over the years
and can be categorized based on the dimension on which
they operate. For example, channel attention considers the
interdependence of the feature maps between channels
and attributes a different weight on each one, while spatial
attention emphasizes interesting regions in the spatial domain.
Popular implementations of the channel attention
mechanism include the squeeze-and-excitation (SE) block
[66] and the efficient channel attention (ECA) [67], while a
spatial attention mechanism commonly used in practice is
the coordinate attention module (CAM) [68]. Several studies
also use a combination of channel and spatial attention,
such as the bottleneck attention module (BAM) [69], the
convolutional block attention module (CBAM) [70] and
the triplet attention [71]. An interesting overview of the attention
mechanisms used in downscaling architectures is
presented in [72].
UPSAMPLING FRAMEWORKS
Although different DL architectures can vary greatly,
four basic downscaling frameworks that describe all approaches
present in the literature can be discerned. These
frameworks are outlined in Figure 5 and represent the
possible ways to design a downscaling DL model with
convolutional and upsampling/downsampling layers as
basic components.
PREUPSAMPLING FRAMEWORK
This is the first framework explored in the literature for image
downscaling via DL approaches. In its most common
form, a traditional upsampling algorithm, e.g., bicubic
interpolation, is utilized to upsample the image to the required
scale. Then, a convolutional neural network (CNN)
model is applied that refines the upsampled image and produces
the HR result. Such an approach provides a simpler
learning pipeline since the network is relieved of the burden
to properly upsample the image and is only tasked to
sharpen and cleanse the input. Another advantage of the
preupsampling framework is the ability to handle images
of arbitrary size and scale. On the other hand, the computational
cost is increased since all operations are performed
in a higher-dimensional space while the preceding upsampling
procedure often amplifies noise and significantly increases
blurring.
(a)
(b)
(c)
(d)
Conv: Convolutional Layer
Upsample: Upsampling Layer
Downsample: Downsampling Layer
FIGURE 5. The possible downscaling frameworks present in the DL literature: (a) preupsampling, (b) Postupsampling, (c) progressive
upsampling, and (d) iterative up- and downsampling. The convolutional, upsampling, and downsampling layers are all trainable. Layers
enclosed by dashed boxes denote stackable blocks.
208
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022
Conv
Conv
Upsample
Upsample
Conv
Conv
Upsample
Conv
Conv
Conv
Conv
Upsample
Downsample
Upsample
Downsample
Upsample
Conv
Upsample
Conv
Conv

IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022