IEEE Geoscience and Remote Sensing Magazine - September 2022 - 204

it (e.g., [29]); these terms are synonymous with upsample
and downsample, respectively. Zhan et al. [30] conducted
research on LST downscaling terminology, among others,
and found that terms such as enhancement, sharpening, fusion,
SR, unmixing, subpixel, and disaggregation are also relevant
to spatial resolution increase. In this article, we use
the term downscale.
DEEP LEARNING FOR REMOTE SENSING
The governing principle of DL is the construction of artificial
neural networks with a large number of layers (indicated
by the adjective deep in the term), which mostly
comprise convolutional, pooling, and fully connected
units. Although several architectures with these building
blocks have been proposed, some of which have been carefully
handcrafted for a specific task, the main idea is the
construction of a hierarchy of features extracted from raw
input data. This hierarchy is computed through representation
learning approaches that can be supervised, semisupervised,
or unsupervised. Overall, the strongest advantage
of DL is its ability to process raw data, thus mitigating the
need for manual feature extraction, and unravel complex
nonlinear dependencies in the input.
One critical factor for the success of any DL method is
the existence of a large and diverse data set to train on. The
abundance and availability of data in EO, therefore, provide
a fertile ground for the application of advanced machine
learning algorithms, and notable progress has been
made over the last decade ([21]-[23]). For example, a number
of works that exploit deeper architectures have recently
been published and achieve impressive results in problems
such as land use and land cover classification [31], scene
classification [32], object detection [33], image fusion [1],
and image registration [34], [35], highlighting the great potential
of DL in RS applications and research.
However, EO poses a unique challenge for DL since it
involves the manipulation of multimodal and multitemporal
data. Remote sensors acquire information from multiple
segments of the EM spectrum, differentiating themselves
from typical CV data, which lie mostly in the red, green,
Training
Upscale/k
Inference
Downscale ×k
blue (RGB) range. In addition, time is quite an important
variable in EO applications. When studying dynamic systems,
information is captured at regular time intervals, and
successive observations must be assessed and compared.
Finally, RS images often suffer from information loss, due
to either hardware failure or atmospheric conditions that
are difficult for certain sensor types to penetrate (e.g., cloud
coverage, haze, and so on are common obstacles for optical
sensors). Therefore, any researcher willing to design and
implement novel DL algorithms for EO must take all of
these points into consideration.
PROBLEM DEFINITION
Given a set of n low-resolution (LR) images (, ,, ),
xx xn12
f
where xX ,
i !
HW and their corresponding high-resolution
#
(HR) images (, ,, ),f where yY , the goal is to
estimate a downscaling function: fX .Y "
yy yn12
i !
:
kH kW
#
Note that H is
the image height, W is the image width, and k is the scaling
factor. This survey presents the approaches that have been
proposed for the estimation of this nonlinear downscaling
function f through deep neural networks.
IMAGING MODEL
The process of obtaining the LR x image from its HR y
equivalent is commonly represented in the literature by the
imaging model
xy bnk
=+7 . () ,
(1)
where b7 is the convolution with a blurring kernel b, k
.
is the downsampling operation by a scaling factor k, and
n is a noise term. This formula is a simple model of the
image degradation taking place during the capture of the
scene and attempts to simulate the physics inside the imaging
sensor. Some researchers have proposed modifications
of this model that account for parameters like the motion
blur, quantization error of the compression process, zooming
effects, exposure time, white balancing, and so on. For a
thorough investigation of the imaging model and its many
extensions, please refer to [14].
FIGURE 2. The Wald's protocol pipeline. The original image (middle) is upscaled by a /k factor,
and the resulting pair is used for model training. The trained model is then transferred to
downscale the original image by a # k factor.
204
WALD'S PROTOCOL
Due to the lack of paired LR-HR images
in most cases, an alternative approach
described by Wald's protocol
[36] is employed (Figure 2). This protocol
assumes that the performance
of data fusion models is independent
of the scale, provided that certain
conditions hold. In their seminal
work, Wald et al. suggest first degrading
the input image according to a
factor k, thus creating LR-HR image
pairs, and proceed to design a model
tasked to downscale it to the original
resolution. Then, the developed
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022

IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022