IEEE Geoscience and Remote Sensing Magazine - September 2022 - 221

method, the authors give, as the input, the gray-scale-transformed
MS image and train the model to learn the colorization
of it. The model learns to generate an original MS
image by taking, as the input, the corresponding reducedresolution
and gray-scale ones. PanColorGAN is trained
using both a reconstruction (MAE) and an adversarial loss.
This can be interpreted as meaning that the model learns
to separate the spectral and spatial components of the MS
image during training.
In conclusion, when hardware and/or time restrictions
apply, L1-RL-FT is a great solution, as it is lightweight and
trains very fast. It also seems to have a good generalization
ability and to solve the problem of insufficient data with
its target-adaptive tuning phase. DML-GMME is a unique
approach that utilizes deep metric learning and autoencoders.
Because it has a rich ablation and is a lightweight model,
a researcher would gain useful insights experimenting
with it. Accurate and sharp results seem to be produced by
LPPNet, a network that simplifies the pansharpening problem
into several pyramid-level learning problems. LPPNet
makes use of the Laplacian pyramid decomposition technique
to decompose the image into different levels that can
differentiate large- and small-scale details, thus achieving
great visual appearance.
Novel ideas that a researcher might want to consider are
presented by Zhang et al. [123] and Luo et al. [114] Zhang
et al. design a special gradient transformation network that
searches the nonlinear mapping between the gradients
of PAN and MS images. Luo et al. propose a PAN -guided
strategy that continuously extracts and fuses features from
the PAN image. VO+Net is a framework that can be put on
top of other approaches to improve the end result. Finally,
SC-PNN is a solution that successfully makes use of saliency
maps and provides great visual results.
HYPERSPECTRAL/MULTISPECTRAL FUSION
HS image sharpening aims at fusing an observable lowspatial-resolution
HS image with a high-spatial-resolution
MS image of the same scene to acquire a HRHS image. One
of the first works to utilize CNNs for HS/MS fusion is introduced
by Palsson et al. in [135], where the authors propose
the use of a 3D CNN with three layers for the HS/MS fusion.
The dimensionality of the HS image is reduced using principal
component analysis to constrain the computational
cost and increase robustness.
Dian et al. [136] propose a deep HS image-sharpening
method called DHSIS that directly learns the priors of the
HRHS image via CNN-based residual learning. They first
initialize the HRHS image by solving a Sylvester equation.
Then, to learn the priors, they utilize the initialized
HRHS image as the input of the CNN to map the residuals
between the reference HRHS image and initialized HRHS
image. This initialization can fully utilize the constraints
of the fusion framework, thus improving the quality of
the input data. The learned priors of the HRHS image are
returned to the fusion framework to reconstruct the final
estimated HRHS image, which can further improve the
performance (Figure 19).
Input Image
Dilated Deformable Convolutional Network
PAN Image
Stacked
Saliency Map
Hybrid Dilated Deformable Block
Saliency Cascade Residual Dense Network
Interpolated
DT-CWT
Detail Map Residual Dense Block
4
MS Image
Spectral Preservation
Convolutional Layer
Deformable Layer
FIGURE 18. An outline of SC-PNN. (Source: [126]; used with permission.)
SEPTEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
221
Dilated Convolutional Layer
Skip Connection
Primary Sharpened Image
Weighted Region
Convolutional
Layer
Saliency
Saliency
Enhancement
Module
Fused Image
IEEE Geoscience and Remote Sensing Magazine - September 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2022
