IEEE Geoscience and Remote Sensing Magazine - September 2022 - 282

in practical applications. Xie et al., in [95], use a strategy
similar to [94] for the task of hyperspectral pansharpening,
also producing promising outcomes. Differing from the
strategy in [94] and [95], new DL network architectures propose
to unfold traditional VO models. In [97], Feng et al.
present a two-step optimization model based on spatial
detail decomposition, then unfold the model under the
gradient descent framework to further construct the corresponding
end-to-end CNN architecture. Similar to [97], Xu
et al., in [98], propose a model-driven deep pansharpening
network by gradient projection. Specifically, two optimization
problems regularized by the deep prior are formulated.
The two problems are solved by a gradient projection algorithm
in which the iterative steps are constructed by two
network blocks that will be effectively trained in an endto-end
manner. Moreover, Cao et al., in [99], and Yin et al.,
in [100], present sparse coding-based strategies to unfold
optimization models into subproblems that are replaced by
learnable networks.
Recently, unsupervised learning strategies have been
introduced into the field of pansharpening; see, e.g., [101]-
[103]. Unsupervised learning explores hidden patterns and
features without any labeled data, which means that there
is no need to simulate data sets with labels for training. It
is a direct approach to network training but strongly dependent
on the effectiveness of the loss function. In [101],
Ma et al. propose a novel unsupervised pansharpening
approach that can avoid the degrading effect of downsampling
high-resolution MS images. The technique also
considers a generative adversarial network (GAN) strategy,
yielding excellent results, in particular, on full-resolution
data. Furthermore, Qu et al., in [103], present a self-attention
mechanism-based unsupervised learning technique
for pansharpening. This can address some challenges, e.g.,
poor performance on full-resolution images and the wide
presence of mixed pixels. In [104], leveraging the targetadaptive
strategy introduced in [74], Ciotola et al. present
an unsupervised full-resolution training framework, demonstrating
its effectiveness on different CNN architectures
[71], [73], [74].
GAN techniques [105] have recently been applied to the
field of image processing. GANs mainly concern learning
generative models via an adversarial process; thus, two
TABLE 1. AN OVERVIEW OF THE PROS AND CONS
OF THE FOUR PANSHARPENING CLASSES.
CS
Spatial fidelity
Spectral fidelity
Generalization ability
Running time
Interpretability
Weak: ; moderate: ; strong:
MRA
VO
ML
models are required to be trained simultaneously, i.e.,
generative models to capture a data distribution and adversarial
models to compute the probability of a sample
belonging to training data. GANs have been applied to
the task of pansharpening; see, e.g., [78], [101], and [106]-
[110]. In [78], Liu et al. utilize a GAN to address the task of
remote sensing pansharpening. This method mainly contains
a two-stream fusion architecture consisting of a generator
to produce the desired HRMS image and a discriminator
to judge whether the image is real or pansharpened.
In [110], to further boost accuracy, the authors propose a
GAN-based pansharpening framework containing two discriminators,
the first dealing with image textures and the
second accounting for image color.
Table 1 gives an overview of the four classes, focusing on
aspects such as spatial fidelity, spectral fidelity, generalization,
running time, and model interpretability. For example,
it is easy to remark that ML methods generally get the
best spatial and spectral performance but require training
and testing data to have similar properties (e.g., a similar
geographic area and acquisition time).
CONTRIBUTION
This article is focused on a deep analysis of the emerging
class of pansharpening algorithms based on ML paradigms.
A complete review of the related literature has been presented.
Henceforth, the article provides a critical comparison of
the state-of-the-art approaches belonging to the ML class.
To this end, a toolbox exploiting a common software platform
and open source ML library for all the ML approaches
has been developed. We would like to stress that this is the
only way to get a critical comparison of ML approaches. In
fact, changing software platforms and ML libraries (e.g.,
TensorFlow and Caffe), results in different built-in functions,
thus generating different behaviors (e.g., a different
initialization of the weights of the network) of the same algorithm
coded in a different environment.
To overcome this limitation, a Python toolbox based
on the PyTorch ML library (which is widely used for applications
such as computer vision and natural language
processing) has been developed. The toolbox will be freely
distributed to the scientific community related to ML and
pansharpening. In this article, eight state-of-the-art approaches
are selected and implemented in the common
framework, following the original implementations proposed
in related papers. A tuning phase to ensure the highest
performance for each approach is performed. This represents
a mandatory step to have a fair comparison because
the eight approaches were originally developed on different
software platforms and using different ML libraries. A
broad experimental analysis, exploiting different test cases,
is conducted with the aim of assessing the performance of
each ML-based state-of-the-art approach.
Widely used sensors for pansharpening are involved
.
[i.e., WorldView-2 (WV2), WorldView-3 (WV3), WorldView-4
(WV4), QuickBird (QB), and Ikonos]. Assessments
282
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2022
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
