IEEE Geoscience and Remote Sensing Magazine - December 2021 - 158

Hughes et al. [11] proposed a pseudo-Siamese CNN for
learning a multisensor correspondence predictor for SAR
and optical image patches. Notably, the networks in [11]
and [58] are trained and validated on the SARptical data set
[182], [183], which is specifically built for the joint analysis
of VHR SAR and optical images in dense urban areas.
In [184], the authors proposed a deep learning framework
that can obtain an end-to-end mapping between image
patch pairs and their matching labels. An image pair is first
transformed into two 1D vectors and then concatenated to
build a large 1D vector as the network input. Then, hidden
layers are stacked for learning the mapping between input
vectors and output binary labels, which indicate their correspondence.
For the purpose of matching SAR and optical images,
Merkle et al. [185] presented a CNN that consists of a
feature extraction stage (a Siamese network) and a similarity
measure stage (a dot product layer). Specifically, features of
input optical and SAR images are extracted via two separate
nine-layer branches and then fed to a dot product layer for
predicting the shift of the optical image within the large SAR
reference patch. Experimental results indicate that this deep
learning-based method outperforms state-of-the-art matching
approaches [186], [187]. Furthermore, Abulkhanov et al.
[188] successfully trained a neural network to build feature
point descriptors to identify corresponding patches among
SAR and optical images and match the detected descriptors
using the random sample consensus algorithm [189]. In
contrast to training a model to identify corresponding image
patches, Merkle et al. [190] first employed a conditional
GAN (cGAN) to generate artificial SAR-like images from optical
images and then matched them with real SAR images.
The authors demonstrated that the matching accuracy and
precision improved through the proposed strategy. Inspired
by that study, more researchers resorted to using GANs for
the purpose of SAR-optical image matching (see [191] and
[192] for a review).
With respect to applications of SAR and optical image
matching, Yao et al. [193] aimed at applying SAR and optical
images to semantic segmentation with deep neural networks.
They collected corresponding optical patches from Google
Earth that accorded to TerraSAR-X patches and built ground
truths using data from OSM. Then, SAR and optical images
were separately fed to different CNNs to predict semantic labels
(buildings, natural areas, land use, and water). Despite
the fact that their experimental results did not outperform the
state of the art [194], likely because of the network design or
the training strategy, they deduced that introducing advanced
models and simultaneously using both data sources can greatly
improve the performance of semantic segmentation. Another
application, mentioned in [195], demonstrated that standard
fusion techniques for SAR and optical images require data
from both sources, which indicates that it is still not easy to
interpret SAR images without the support of optical ones. To
address this issue, Schmitt et al. [195] proposed an automatic
colorization network composed of a VAE and a mixture density
network [196] to predict artificially colored SAR images (i.e.,
Sentinel-1 images). These images proved to disclose more information
to human interpreters than the original SAR data did.
In [42], the authors tackled the problem of cloud removal
from optical imagery. They introduced a cGAN architecture
to fuse SAR and cloud-corrupted multispectral data for
generating cloud- and haze-free multispectral optical information.
Experiments proved the effectiveness of the proposed
network for removing clouds from multispectral data
with auxiliary SAR data. Extending previous multimodal
networks for cloud removal, the authors of [43] proposed a
cycle-consistent GAN architecture [197] that utilizes an image
forward-backward translation consistency loss. Cloudcovered
optical information is reconstructed via SAR data
fusion, while changes to cloud-free areas are minimized
through the cycle consistency loss. The cycle-consistent architecture
facilitates training without pixelwise correspondences
between cloudy input and cloud-free target optical
imagery, relaxing requirements for the training data set.
In summary, it can be seen that the utilization of deep
learning methods for SAR-optical data fusion has been
a hot topic in the remote sensing community. Although
a handful of data sets consisting of optical and SAR corresponding
image patches is available for different terrain
types and applications, one of the biggest problems remains
the scarcity of high-quality training data. Semisupervised
methods, as proposed in [198], seem to be a viable option
(a)
(b)
(c)
(d)
FIGURE 9. Randomly selected patches obtained from the testing phase of the network for SAR-optical image patch correspondence detection
proposed in [11]. (a) True positives. (b) False positives. (c) False negatives. (d) True negatives.
158
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2021

IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021