IEEE Geoscience and Remote Sensing Magazine - June 2022 - 244

FUSION USING THE MODEL-CONSTRAINED
LOSS FUNCTION APPROACH
In the model-constrained, loss function approach, Zhang
et al. [48] proposed an unsupervised deep framework for
blind HSI superresolution. The loss function in [48] is
min ZXPY Xk kP (( )
X =f (),
H Pk
s.t.
, ,
-+ -+ +
H E
)
m
2
where P RsS
!
#
22 22
) . r
F
(29)
domain, k denotes the unknown blur kernel, r
denotes the degeneration in the spectral
. indicates
the downsampling operation in the spatial domain with
scaling factor ,r and ) indicates the blurring operation in
the spatial domain. The first two terms correspond to the
spatial fidelity and spectral fidelity terms, respectively.
The third term is the regularization term that imposes
constraints on the degradation processes, that is, k and
P , which guides the network to estimate the degradations,
and improves the ability of the fusion network to comply
with the unknown degenerations in real HSI superresolution
applications. H-parameterized ()f $
H
represents
the image generator network for the latent X , and E is a
precomputed code that contains the image-specific statistics
of
X .
ALTHOUGH THE COUPLED
MODEL- AND DATA-DRIVEN
METHODS HAVE SHOWN
THEIR POTENTIAL IN
LEARNING THE NONLINEAR
COR RELATIONS IN REMOTE
SENSING IMAGERY AND
CHARACTERIZING THE
STRUCTURAL THIS INSIGHT
OF OPTIMIZATION.
Integrating the observation model into the loss
function construction reduces
the need for ideal images
as labeled data for network
training, which is very practical
and suitable for situations
where limited sample
data are available.
In general, in remote sensing
image fusion, the dataand
model-driven cascading
approaches and the variational
model with embedded
learning approach are simple
and easy to implement. The
model-constrained network
structure approach shows good
performance due to its interpretable
structure, but it needs careful mathematical derivation
and network structure design. Therefore, there is still
not much related work in remote sensing image fusion. Due
to the low dependence on ideal, high-quality images as labeled
data, the model-constrained, loss function approach
has been used in some unsupervised fusion schemes recently,
and it has the potential to be joined with the other
three combined approaches.
FURTHER DEVELOPMENTS
Although the coupled model- and data-driven methods
have shown their potential in learning the nonlinear correlations
in remote sensing imagery and characterizing the
structural this insight of optimization based on the image
degradation process, challenges still exist due to the
244
complexity of physical information, availability of samples,
and universality of the models. In the following sections,
we suggest a few future directions.
STRUCTURE DEVELOPMENT
INTRODUCTION OF GRAPH NEURAL NETWORKS
Two aspects limit the development of combined strategies
in remote sensing applications. First, the convolution
operation underpinning all CNN architectures is unable
to capture nonlocal, self-similarity patterns because of
the locality of convolution kernels. Furthermore, the objects
or pixels in multimodal data, including optical data,
infrared data, SAR data, lidar data, and socioeconomic
data, contain complex relationships and interdependencies.
The complexity of the data has imposed significant
challenges on existing CNN architectures. Graph neural
networks (GCNs) are able to describe the complex relationships
between data by the nodes and edges in the
feature space of the network [49]-[51] and have shown
great potential for capturing self-similarity information
and coping with heterogeneous multimodal data. Very recently,
a graph convolutional denoiser network [52] was
proposed to aggregate k-nearest neighboring patches for
image denoising. In addition, by further mining the multiscale
recurrence property of a natural image, cross-scale
internal GCN [53] was proposed to construct a graph
on different-resolution patches and successfully recover
more detailed textures. However, due to the big difference
between CNNs and GCNs, when the data are structured
as a graph, the combination of iterative optimization and
GCNs still faces big challenges. First, graph-based models
are less efficient than CNN-based ones. The graph-based
models usually need to handle the whole image, with
complex topology structures containing a large number
of nodes. For each iteration in the optimization, the
graph should be updated with changes of an optimal solution.
Hence, graph convolutions need to adapt to the
dynamicity of the graphs. The change of adjacency relations
for each node, however, may also pose a burden to
computational efficiency and introduce turbulence into
network optimization.
UTILIZATION OF UNSUPERVISED LEARNING
Most of the existing data-driven image restoration methods
are based on supervised learning, i.e., they utilize the
ideal clean images as labeled data and train the network
through a large number of samples. However, in actual situations,
the following two problems may be encountered.
One is that there may be no ideal clean images as labeled
data of networks, such as clean HSIs or PolSAR images
in the denoising. The other is the insufficient number of
sample data pairs, for example, due to the difficulty in
obtaining pairwise hyperspectral and multispectral images
at the same time. Therefore, scholars have to degrade
the observations and use the original observations as
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE JUNE 2022
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
