IEEE Geoscience and Remote Sensing Magazine - September 2022 - 8

Moving from applications and instrument
techniques
IN CONTRAST TO " DATA
LAKES, " WHICH SERVE AS
FLEXIBLE DATA STORES FOR
CLOUD COMPUTING, " DATA
SWAMPS " HAVE LITTLE
ORGANIZATION, DATA
CURATION, AND ACTIVE
MANAGEMENT.
on to algorithms, Arienzo et al. present a review of quality
protocols and indexes for pansharpening (PS). This is the
combination of a multispectral image with a panchromatic
image at a higher resolution to produce a fused image with
both high resolution and spectral information. The quality
assessment of the resulting PS image is not a trivial issue, as
there is no reference image to compare with, and quantifying
the quality of a PS image is an ill-posed problem. The article
begins with a thorough review of quality protocols based
upon full-resolution images before moving to commonly
used quality indexes related to the " quality with no reference "
protocol. MATLAB code
implementations of these protocols
are freely distributed to
the scientific community. The
consistency and reversibility
of the PS process are also considered.
The spatial-spectral
quality paradox, in which
the least spectrally distorted
image would result from no
injection of spatial information,
is introduced. Quality
assessment is tested on three
satellite datasets representing
different spatial resolution ratios, numbers of spectral
bands, and landscapes, processed with a benchmark collection
of 30 state-of-the-art PS methods, some of which
were selected for the cover image of this issue. The authors
provide extensive tables summarizing comparisons among
the different algorithms.
Remote sensing data are characterized by a number of
spatial, temporal, and spectral scales, and the integration
and comparison of data having different scales is an important
but sometimes difficult task. PS, as we have just seen,
is one approach to optimize the combination of data having
different scales. Downscaling, the transition from low
to high spatial resolution, is another common technique
for changing the scale of remote sensing images. Sdraka
et al. present a review of deep learning for downscaling.
They present a comprehensive taxonomy of the various
downscaling methods and derive some basic theory. A
study of the most popular metrics for evaluating downscaled
images and an analysis of the tradeoff between perception
and distortion (which can be shown to act in both
complementary and competitive ways) is also presented.
Although similar problems exist in the computer vision
community, Earth observation data have quite different
characteristics, including the combination of multiple bands
across the electromagnetic spectrum and distortion due
to the atmosphere. Nonetheless, the authors point out the
value in studying recent advances in computer vision for
inspiration, providing references to several recent review
articles in that field and summarizing some promising new
ideas that have not yet been applied to remote sensing.
8
Deep learning, in all applications, requires a good training
dataset. This presents a problem in many areas of remote
sensing. There are few public benchmark datasets
available for the evaluation of downscaling methods. This
review did identify a large number of datasets provided for
other remote sensing problems that may be applicable, in
a limited way, to test downscaling. In addition to a careful
review of related work in computer vision and the need for
benchmark datasets, the authors point out several promising
directions for future research, including the need to
derive universal metrics, improved model interpretability,
explainable artificial intelligence, generational adversarial
networks, unsupervised learning methods, and the extension
of these methods to SAR. As in most other fields, open
source code and reproducibility are desired.
The selection of labeled training data is also very important
in remote sensing applications. Active learning (AL)
methods, reviewed by Thoreau et al., optimize the annotation
of training datasets. This can be particularly important
in cases in which much of the information is contained
in rarely occurring samples, typically in change detection
and object detection problems. AL techniques can be a
cost-effective method to select a representative training set.
This article aims to present some insights and guidelines
for choosing an appropriate AL technique in practical use
cases. The scope of this review is " pool-based AL, " in which
a query system identifies samples with the highest information
content and an oracle system produces labels for
these samples. Three classes of AL are covered: uncertainty
based, representativeness based, and performance based.
Six state-of-the-art methods spanning each of these classes
are demonstrated and evaluated using a toy dataset, a classic
hyperspectral scene, and a complex hyperspectral scene.
A toolbox has been released so that readers can continue
these experiments on their own.
Our final feature returns to PS. Deng et al. review recent
developments in machine learning (ML) for this application,
comparing eight different approaches based around
a common software platform and open source library.
For comparison, they also review some classical methods:
component substitution and variational optimization. The
article provides an overview of the mathematical theory
for all the methods considered. ML approaches include
dictionary learning methods and compressed sensing, but
recently, deep learning has begun to find wide applications
in almost all areas of remote sensing. Integrating ML with
classical methods and variational optimization is a promising
new direction. The performance of these methods is
compared using well-known protocols applied to data from
the Worldview-2, 3, and 4; Quickbird; and Ikonos satellites. The
generalization of the method with changes in the instrument,
improvement with fine-tuning, and computational
performance are also analyzed. The authors conclude with
some guidelines for future development of new PS methods
based on ML, including the design of multiscaled architectures,
fine-tuning to increase the generalization ability,
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
