IEEE Geoscience and Remote Sensing Magazine - March 2021 - 62

where B contains the PC scores corresponding to Y and G is
an r × q matrix corresponding to Z, with r being the number of pixels of the PAN image and q the number of PCs
used in the model. The pansharpening is based on solving
the maximum a posteriori problem
gt = argmax p (g|b, y PAN), (19)

	

where g and b represent the wavelet-transformed and waveletvectorized matrices G and B. By making certain assumptions, such as the spatial independence of wavelet coefficients, an efficient algorithm can be derived. This algorithm
is detailed in [79].
REDUCED-RANK-BASED PANSHARPENING
The main idea behind reduced-rank pansharpening is to
exploit the spectral correlations between the MS and PAN
images. This is done by defining a matrix, X, that has the
PAN image in its first column and the pansharpened MS
images in column 2 to N + 1. We then assume that it has a
low-rank representation,
X = [x i] iN=+11 = GF T, (20)

	

where G = [g j] lj = 1, F = [f Ti ] li = 1 is an orthonormal matrix
and l 1 N + 1 is the low rank. The individual bands can
be retrieved from this low-rank representation as x i = Gfi .
The pansharpening problem is then to minimize the following cost function:
	

J (F, G) =

N+1

/

i=1

1
2 y i - M i B i Gfi

2

l

+

/ m j z w (g j),(21)

j=1

where B i and M i are the blurring and decimation matrix corresponding to band i, respectively; m j 2 0 are the
weights; and z w (g j) is a weighted roughness penalty further described in [134]. The cost function is minimized
by using a cyclic descent algorithm that alternates between minimizing with respect to F and G. The optimization problem with respect to G is a convex problem
and is solved efficiently by using conjugate gradient.
The orthonormality-constrained optimization problem
with respect to F is efficiently solved by using manifold optimization [135], [136]. Further details are given
in [72], [134].
MACHINE LEARNING
Nowadays, among ML approaches, CNN-based methods
are the most promising ones. Therefore, we focus here on
a family of solutions rooted in the pioneering work of Masi
et al. [91], among the first to use CNNs for pansharpening.
In particular, we present four models, two of which, hereinafter referred to as PNN and PNN-IDX (where IDX stands
for the use of some indexes as auxiliary inputs), were introduced in [91], and the other two, advanced versions of
PNN, here named A-PNN and A-PNN-FT (where FT stands
for fine-tuning), were proposed more recently in [95].
62

The development of a DL method for pansharpening involves at least the following steps:
1)	 training data set setup
2)	 network architecture design
3)	 definition of a proper loss function to guide the training
4)	 network training using a proper optimization method
5)	 network validation on a dedicated data set.
As a peculiar trait, all CNN-based methods for pansharpening share a common key problem, which is related to the
collection of data for training (step 1), i.e., a sufficiently
" rich " set of input-output examples to learn the unknown
network parameters (say, U). Ideally, pansharpened images
are, in fact, unavailable and therefore must be synthesized
using an ad hoc generation procedure. Figure 3 depicts the
conceptual scheme of the training process proposed in [91].
The underlying idea is a resolution shift, such that the MS
component plays the role of reference output rather than
input. This is achieved by properly downgrading the resolution (dashed section) of both MS and PAN components of
any training image with a resolution ratio of R. Then, an interpolation of the RR MS (MS .) back to its original size provides a properly sized component (R
MS .) to be concatenated
with the downgraded PAN image (P.) prior to feeding the
CNN. In practice, this process is repeated on one or more
images reserved for training and combined with a cropping process. By doing so, we come to have thousands of
input-output pairs in an RR domain. This allows organizing the training data in minibatches for efficient iterative
optimization. In particular, at iteration n the pansharpening outcome \
MS . is compared to the corresponding reference (original MS) using a suitably defined loss function,
L ($ , $). Taking the gradient of the loss, eventually the optimizer computes the updated CNN parameter set U n + 1 . All
models presented here make use of a stochastic gradient
descent algorithm with momentum. The resolution downgrading process, carried out according to the sensor MTF,
soon became a standard option for CNN-based pansharpening methods after its introduction in [91], to which interested readers are referred for further details. Once the training process is completed, the network is ready to be used on
test images, as summarized in Figure 4 for the two variants
proposed in [91], PNN and PNN-IDX (including the dashed
section and by properly training the CNN, also considering
these new inputs).
Network architecture design (step 2) is probably the item
that most characterizes the different proposals. In general
terms, it amounts to a definition of a directed acyclic graph
(DAG), which describes the input-output information flow,
associating a specific task (such as convolution, point-wise
nonlinearities [e.g., rectified linear unit (ReLU)], batch normalization, concatenation, sum, and pooling) to each DAG
vertex. Specific subgraph structures obtained by combining
these elementary operations are also often employed (e.g.,
residual, dense, or inception modules). The PNN model
[91] is a relatively simple one: a serial net composed of three
convolutional units interleaved by ReLU activations. The
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

MARCH 2021
IEEE Geoscience and Remote Sensing Magazine - March 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - March 2021
