IEEE Geoscience and Remote Sensing Magazine - June 2022 - 316

only two typical applications that are directly related
to image fusion with GANs: missing-data reconstruction
and thin cloud removal. Usually, there are some
overlaps between missing-data reconstruction, spatiotemporal
image fusion, cloud removal, and so on. To
be more clear, we chart their relationships in Figure 20.
We can observe that, for some applications, they can be
realized by both homogeneous and heterogeneous RS
image fusion.
MISSING-DATA RECONSTRUCTION
There are often missing data in RS images because of defective
sensors, shadows, thick clouds, and so on, so the value
and application of the acquired RS data are greatly reduced
(Figure 21). Image fusion is one of the most popular methods
to reconstruct the missing data. As in [188], the methods
for missing-data reconstruction in RS images can be
divided into four categories: spatial-based, spectral-based,
temporal-based, and hybrid methods. Both homogeneous
RS data and heterogeneous data can be fused to reconstruct
missing data.
Before introducing a GAN into missing-data reconstruction,
there are already some studies that are based on deep
learning, such as CNNs. For example, in [189], a unified
spatiotemporal-spectral framework based on a deep CNN
was proposed for missing information reconstruction; it can
be applied to deadlines in Aqua MODIS, the Landsat ETM+
Scan Line Corrector (SLC)-off problem, and thick cloud removal.
It is not necessary to use image fusion for missingdata
reconstruction. Without a GAN or fusion, it belongs to
a simple and direct problem of data reconstruction, such as
in [190], in which the authors translated incomplete data
into their corresponding complete data. There are some
overlaps between missing-data reconstruction, spatiotemporal
fusion, and SAR-optical fusion, which are illustrated
in Figure 20.
In RS, the key problem for missing image reconstruction
is how to integrate the auxiliary information (spatial,
spectral, temporal, and so on) or priors into the
reconstruction by current generators. However, unlike
spatial-spectral or spatiotemporal fusion, missing data
are often large or irregular areas but not common downsampling.
Please refer to the observation about (12) that
the rank of matrix H for missing data is often lower
than that in other fusion problems. It means that the
ill-posedness of the equation is more serious from the
viewpoint of solving an inverse problem. Without more
auxiliary information from fusion, we usually can reconstruct
only small areas.
Implicit, data-driven methods, such as CNNs or U-nets
can improve the performance on the large missing area, but
they still play a limited role. A GAN opens up new prospects
for missing-data reconstruction. Part of its ability
comes from its schema of data generating. In [191], the authors
summarize a common model of GAN-based missingdata
imputation: the generator's goal is to accurately impute
316
missing data, and the discriminator's goal is to distinguish
between observed and imputed components. The discriminator
is trained to minimize the classification loss, but the
generator is trained to maximize the discriminator's misclassification
rate.
There are substantial studies on missing-data reconstruction
in the field of computer vision that are known as image
inpainting. Inpainting is more like a spatial-based method.
In RS applications, more reference information or priors
can be utilized in missing-data reconstruction. Therefore,
there are often two or more inputs in reconstruction, so it
can be taken as an image fusion problem. For simplicity, we
categorize them into small-/medium-scale and large-scale
missing-data reconstruction.
The term small- and medium-scale missing data mainly
refers to the missing-data phenomena caused by defective
sensors [189]; orbit gaps [192], [193]; and temporal
gaps [194]. In most cases, the thick clouds are not small-/
medium-scale missing data but large-area missing data.
The missing data of voids in mountainous areas by the
Shuttle Radar Topography Mission are small- and medium-scale
missing data. The authors in [195] incorporated
the shadow geometric constraints into the CGAN. The
shadow boundary, shadow ceiling, and shadow entrance
curvature loss functions are combined with an adversarial
loss to guide the generator to well predict the value in
void areas.
The missing data in comprehensive traffic flow data
[196] are small-scale missing data. In [196], the authors
proposed a spatiotemporal learnable bidirectional attention
GAN to implement data fusion for missing traffic
data imputation. In that study, the masked reconstruction,
perceptual, discriminative, and adversarial losses
were combined as a new objective function and optimized
to improve the data imputation ability. In some
applications, the data collection process is likely to generate
data with missing, incomplete, or corrupted modalities.
In [123], the authors focused on the semantic
segmentation of building footprints with missing modalities
in which GANs are effectively used to synthesize
the missing or incomplete data in the depth map.
The term large-scale missing data mainly refers to the
missing-data phenomena caused by clouds [201], shadows,
and large-area occlusion [198]. To deal with very-high-resolution
images with large-scale missing regions, [198] proposed
that the reconstruction process be divided into two
connected parts: structure prediction and texture generation.
In the first part, one generator predicts the edges of objects
in missing regions; then, in the second part, the other
generator predicts the textures based on the edge structural
information from the first part. There are two cascaded
generators and one discriminator in this method, and it is
a spatial-based method without any auxiliary spectral or
temporal data.
Thick cloud removal is one of the important subbranches
of large-scale missing-data reconstruction. In
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE JUNE 2022
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
