IEEE Geoscience and Remote Sensing Magazine - December 2021 - 162

SEN1-2
The SEN1-2 data set [203] includes 282,384 pairs of corresponding
Sentinel-1 single-polarization-intensity and Sentinel-2
RGB image patches collected from across the globe
and in all meteorological seasons. The patches are 256 ×
256 pixels. Their distribution through the four seasons is
roughly even. SEN1-2 is the first large open data set of this
kind. We believe it will support further developments in
the field of deep learning for remote sensing as well as multisensor
data fusion, such as SAR image colorization and
SAR-optical image matching.
OTHER DATA SETS
SAMPLE PolSAR IMAGES FROM
THE EUROPEAN SPACE AGENCY
These data sets (https://earth.esa.int/web/polsarpro/data
-sources/sample-datasets) include, for example, the Flevoland
PolSAR data set, which several works use for agricultural
land use/land cover classification. The authors of
[216]-[218] manually labeled it according to different classification
schemes.
SAR IMAGE LAND COVER
This data set [219] is not publicly available. Readers should
contact the creator.
AIRBUS SHIP DETECTION CHALLENGE
This data set can be accessed at https://www.kaggle.com/c/
airbus-ship-detection.
CONCLUSION AND FUTURE TRENDS
This article reviewed the state of the art of an important
and underexploited research field: deep learning in SAR.
Relevant deep learning models were introduced, and their
applications in six application fields-terrain surface classification,
object detection, parameter inversion, despeckling,
InSAR, and SAR-optical data fusion-were analyzed
in depth. Existing benchmark data sets and their limitations
were discussed. In summary, despite early successes, the full
exploitation of deep learning in SAR is mostly limited by 1)
the lack of large and representative benchmark data sets and
2) the defect of tailored deep learning models that makes
full consideration of SAR signal characteristics difficult.
Looking forward, the years ahead will be exciting. Nextgeneration
spaceborne SAR missions will simultaneously
provide high-resolution and global coverage, which will
enable novel applications, such as monitoring the dynamic
Earth. To retrieve geoparameters from these data, the development
of new analytics methods is warranted. Deep
learning is among the most promising methods. To fully
unlock its potential in SAR/InSAR applications in this big
SAR data era, there are several promising future directions,
including the following:
◗ Large and representative benchmark data sets: As summarized
in this article, there is only a handful of SAR
162
benchmarks, in particular, when multimodal ones are
excluded. For instance, in SAR target detection, methods
are mainly tested on a single benchmark data set,
MSTAR, where only several thousands of target samples
(several hundred for each class) are provided for training.
With respect to InSAR, due to the lack of ground truth,
data sets are extremely deficient or nearly nonexistent.
Large and representative expert-annotated benchmark
data sets are in high demand in the SAR community and
deserve more attention.
◗ Unsupervised deep learning: To bypass the deficiencies in
annotated data in SAR, unsupervised deep learning is
a promising direction. These algorithms derive insights
directly from the data themselves and work as feature
learning, representation learning, and clustering, which
could be further used for data-driven analytics. Autoencoders
and their extensions, such as VAEs and deep embedded
clustering algorithms, are popular choices. With
respect to denoising, in despeckling, the high complexity
of SAR images and the lack of ground truth make it
infeasible to produce appropriate benchmarks from real
data. Noise to noise [152] is an elegant example of unsupervised
denoising, where the authors of [152] learn denoised
data without clean data. Despite the nice visual
appearance of the results, preserving details is a must for
SAR applications.
◗ Interferometric data processing: Since deep learning methods
were initially applied to perception tasks in computer
vision, many methods resort to transforming SAR
images, e.g., PolSAR images, into RGB-like images in advance,
or they focus only on intensities. In other words,
the most essential component of an SAR measurement-
the phase information-is not appropriately considered.
Although CV-CNNs are capable of learning phase information
and show great potential in processing CV-SAR
images, only a few such attempts have been made [83].
Extending CNNs to the complex domain, while preserving
precious phase information, would enable networks
to directly learn features from raw data and would open
up a wide range of SAR/InSAR applications.
◗ Quantification of uncertainties: Generally speaking, geoparameter
estimates without uncertainty measures are considered
invalid in remote sensing. Appropriately trained
deep learning models can achieve highly accurate predictions.
Yet they fail in quantifying the uncertainty of these
predictions. Here, giving a statement about the predictive
uncertainty, while considering both aleatoric uncertainty
and epistemic uncertainty, is of crucial importance.
The Bayesian deep learning community has developed
a model-agnostic and easy-to-implement methodology
to estimate both the data uncertainty and model uncertainty
within deep learning models [54], which is awaiting
exploration by the SAR community.
◗ Large-scale nonlinear optimization problems: The development
of inversion algorithms should keep up the pace
of data growth. Fast solvers are demanded for many
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2021
https://earth.esa.int/web/polsarpro/data-sources/sample-datasets https://earth.esa.int/web/polsarpro/data-sources/sample-datasets https://www.kaggle.com/c/airbus-ship-detection https://www.kaggle.com/c/airbus-ship-detection
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
