IEEE Geoscience and Remote Sensing Magazine - March 2021 - 56

authors of [67] assumed A1 and A2 but employed a TV regularizer, separately, on each band of the HRMS images. In
[68], information about the PAN image was incorporated
into the TV regularizer. Furthermore, a low-rank structure
was assumed in the spectral domain. The authors of [69]
relaxed A2 by imposing it only for high-frequency components and used a roughness that includes both the HRMS
images and the PAN image. The authors of [70] and [71]
share a regularizer with the P + XS method but propose different fidelity terms. The authors of [72] did not directly assume A2 but indirectly modeled the relationship between
the PAN and the HRMS images by considering a spectral
low-rank relationship between them, using an l 2 roughness
penalty to regularize the solution. In [50], an algorithm is
proposed to directly estimate the spatial relationship between the PAN and the MS images. Other notable articles
include [73]-[76].
Bayesian methods also fall into the VO category. The
authors of [77] assumed a joint Gaussian model for the
unknown MS image and the PAN method; [78] extended
this idea by incorporating wavelet transformation into the
formulation; [79] further extended this model using PCA to
enforce a low-rank spectral structure. Other notable Bayesian articles include [76] and [80].
Another class of methods belonging to the VO family relies on sparse representations [81]. The idea is to represent
the unknown HRMS image as a sparse linear combination
of dictionary elements (here, image patches). The main
challenge is the generation of the dictionary, with contributions ranging from using available PAN and LRMS images
[82] to using the data at hand [83]-[87] to create the dictionary. In [88], sparse representation was used to reconstruct
only the high-resolution details to be injected into LRMS.
Most of the VO methods described previously depend on
one or more regularization parameters. These parameters
need to be selected by the user. This critical problem is mostly unexplored in the pansharpening literature, and most
articles report what settings were used in the experiments.
However, some articles provide guidelines, e.g., [72], [86].
MACHINE LEARNING
ML has has moved its early steps toward pansharpening in
the last decade. Pioneering works in this category include
compressive sensing or dictionary-based solutions, such as
[81], [83]-[86], [88]. Needless to say, the deep learning (DL)
wave has even flooded over into the remote sensing field
[19], including pansharpening [89]-[100] and intimately
related tasks, such as superresolution [101]-[103] or HS/
MS data fusion [104], [105]. The first attempt by Huang
et al. in 2015 relies on a modified sparse denoising autoencoder scheme [89]. In 2016, Zhong et al. [90] instead proposed a CS approach based on the GS transform integrated
with an off-the-shelf superresolution convolutional neural
network (CNN) [106] to upsample the MS component. In
parallel, in 2016, Masi et al.[91] designed and trained the
first fully convolutional network specifically conceived for
56

pansharpening, the pansharpening neural network (PNN),
achieving state-of-the-art results. Motivated by the encouraging results of the PNN, many researchers have moved
on to this line of inquiry. In particular, the use of CNNs
with residual learning modules [107] will soon become a
standard option for pansharpening, as testified to by many
subsequent works [92]-[100]. In contrast to the relatively
simple architecture of the PNN and its advanced version,
advanced PNN (A-PNN) [95], other works explore more
complex models in terms of depth [92], [94], [99] and/or
topology [97], [98]. Different training strategies have also
been investigated by means of different target loss functions, such as the mean squared error (MSE) [91], 92, [96]-
[98], the mean absolute error [95], [99] or the erreur relative
globale adimensionnelle de synthése (ERGAS) [100], also using
adaptive tuning schemes (A-PNN) or resorting to the generative adversarial learning paradigm [99].
A common trait of these ML approaches is the assumption of a training paradigm relying on a resolution downgrade process (e.g., Wald's protocol) to provide labeled data
for supervised training. By doing so, due to the scale sensitivity of the pansharpening problem, spatial consistency
was soon recognized as a primary issue for ML methods. In
fact, the latest ML solutions seek to address such a problem
using different training paradigms, in particular, by means
of multiobjective strategies. In [108] a joint reduced-/fullscale loss is proposed to enforce spatial fidelity thanks to
the introduction of an additional loss term computed at the
target scale. Another interesting example is the Pan-GAN
model [109], which relies on spectral-spatial multiobjective adversarial training using two dedicated discriminators. It is also worth mentioning other HS/MS fusion approaches, staightforwardly extendable to pansharpening,
that include unsupervised training schemes to avoid the
resolution downgrade of the data to be used for training
[104], [105].
ASSESSMENT
Quality assessment of MS pansharpening has been the object of extensive research activities. Since the target of image
fusion is unavailable, several protocols of quality evaluation have been developed to overcome the lack of a reference image.
A widely adopted protocol was first proposed in [110]
and further discussed in [111] and [112]. Such a protocol
relies on two properties that the fused data have to satisfy,
referred to as consistency and synthesis, which are reviewed
in the " Quality Assessment of Fusion Products " section,
together with the indexes that can be adopted for the quantitative assessment of such properties. It should be noted
that the consistency property can be checked by comparing
the original MS image to a degraded version of the pansharpened image obtained through modulation transfer
function (MTF) decimation filters tuned to the MS sensor. This property, which represents a necessary condition
to be satisfied by any pansharpening algorithm, has been
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

MARCH 2021
IEEE Geoscience and Remote Sensing Magazine - March 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - March 2021
