IEEE Geoscience and Remote Sensing Magazine - September 2023 - 92

actually sharing very similar content. For example, a high
level of detail in class semantics and terminology makes it
difficult to compare the reference data of different datasets.
A typical example is land cover classification, where similar
classes may be aggregated into different subcategories
depending on the application. As a result, models trained
on different application-specific datasets may actually approximate
very similar functional relationships between
image data and target variables.
Virtually all of the less recent and still most of the modern
datasets aim for specificity. However, several of the
more recent benchmarks follow another direction: generality,
i.e., providing more sensor modalities than actually
required plus large-scale, often noisy reference data for
multiple tasks instead of small scale and carefully curated
annotations that only address a single task.
The contribution of such general datasets is manifold:
first and foremost, the required number of (annotated)
training samples for fully supervised ML simply does not
scale very well given the effort of data annotation and curation
in remote sensing. Thus, such general, large-scale
datasets introduce new factors that increase the relation
to realistic application scenarios such as robustness to
label noise (e.g., by leveraging existing semantic maps
as reference data, which are often outdated, misaligned,
or of coarse resolution) and weakly supervised learning
(where the reference data have a lower level of detail than
the actual target variable, e.g., training semantic segmentation
networks with labels on image level). Large-scale
datasets are the only option to realistically test the generalization
capabilities of learning-based models, e.g., over
different geographic regions, seasons, or other domain
The Ideal Pretraining Dataset
A dataset ideally suited for pretraining and/or self-supervised learning should
adhere to as many of the following characteristics as possible:
◗ multiple platforms (vehicle, drone, airplane, and satellite)
◗ multiple sensors (Planet, SPOT, WorldView, Landsat, Sentinel 1/2, and so
forth)
◗ several acquisition modalities (SAR, RGB, hyperspectral, multispectral, thermal,
lidar, passive microwave, and so on)
◗ diverse acquisition geometries (viewing angles, e.g., off-nadir conditions and
spatial and temporal baselines in multiview data, e.g., interferometric SAR)
◗ realistic distortion factors (cloud cover, dust, smog, fog, atmospheric influence,
spatial misalignments and temporal changes in multiview data, and so forth)
◗ well distributed geographical locations (spatial distribution within the dataset,
climate zones, socioeconomic and cultural factors, and different topographies)
◗ diverse land cover/use (urban, rural, forest, agricultural, water, and so on)
◗ varying spatial resolution (0.1-1 m, 3-10 m, 10-30 m, 100-500 m, and scale
distribution)
◗ temporally well distributed (seasonality, lighting condition, sun angle, and
nighttime imagery)
◗ a diverse set of reference data that are well aligned with the EO measurements
(semantic information, change, geo/biophysical parameters, and so forth).
92
gaps. Furthermore, although multimodal datasets enable
data-fusion and cross-modal approaches that leverage the
different input sources, multitask datasets allow exploiting
the mutual overlap of related tasks regarding feature
extraction and representation learning. Finally, the idea
of loosening the previously tight relationship between
input data and the target variable in datasets (up to the
point where a dataset might not offer reference data for
any target variable) is to provide data that can be leveraged
to learn powerful representations that are useful for
a large variety of downstream tasks (as in pretraining or
self-supervised learning).
However, there is not yet a single " go-to " dataset that can
be used for pretraining most of the newly developed models
or for benchmarking specific tasks against state-of-theart
approaches. Collecting such a high-quality benchmark
dataset that enables pretraining of models for as many
downstream tasks as possible is of significant value for further
pushing performance boundaries.
Figure 28 presents a schematic diagram of the properties
of an ideal solution for a go-to EO benchmark dataset,
covering diverse geolocations, multiple modalities,
different acquisition scenarios, and various applications.
It is ideally acquired by different types of sensors and
platforms with different viewing geometries to cover
objects from different look angles. The images are obtained
from different electromagnetic spectrum bands,
i.e., visible, infrared, thermal, and microwave, resulting
in multi-/hyperspectral, SAR, lidar, optical, thermal, and
passive microwave measurements. The reference information
or annotations are provided on a level that allows
defining various tasks based on a single annotation. For
example, an image with dense semantic annotations allows
users to generate their desired object instance annotation
files. Extending the dataset to multiple images of
a scene with corresponding semantic labels enables not
only semantic segmentation but also semantic CD tasks.
In summary, we foresee a certain duality in the future development
of EO-related datasets: on the one hand, following
the paradigm of data-centric ML [44], i.e., moving against
the current trend of creating performance gains merely by
leveraging more training data but instead focusing on datasets
tailored toward specific problems with well-curated, highquality
reference data (e.g., manually annotated or based on
official sources). On the other hand, general datasets that cover
as many input and output modalities as possible to allow
learning generic representations that are of value for a large
number of possible downstream tasks.
FINDABILITY, ACCESSIBILITY, INTEROPERABILITY,
REUSE AND ARD
In addition to the content, scope, and purpose of datasets,
their organization will gain importance. With only a
dozen public datasets available prior to 2015, it was feasible
that each is provided with its own data format and
meta-information, hosted on individual web pages, and
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2023
IEEE Geoscience and Remote Sensing Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2023
