IEEE Geoscience and Remote Sensing Magazine - September 2023 - 64

large-scale datasets to reach the desired accuracy and generalization
performance. Therefore, the availability of annotated
datasets has become a dominating factor for many
cases of modern EO data analysis that develops and evaluates
powerful, DL-based techniques for the automated interpretation
of remote sensing data.
The main goal of general computer vision is the
analysis of optical images, such as photos, which contain
everyday objects, e.g., furniture, animals, or road
signs. Remote sensing involves a larger variety of sensor
modalities and image analysis tasks than conventional
computer vision, rendering the annotation of remote
sensing data more difficult and costly. Besides classical
optical images, multi- or hyperspectral sensors and
different kinds of infrared sensors; active sensor technologies
such as laser scanning, microwave altimeters,
and synthetic aperture radar (SAR) are regularly used,
too. The fields of application range from computer
vision-like tasks, such as object detection and classification,
to semantic segmentation (mainly for land cover
mapping) to specialized regression tasks grounded in the
physics of the used remote sensing system.
To provide an illustrative example, a dataset for biomass
regression from interferometric SAR data will adopt
imagery and annotations very different from the ones
needed for the semantic segmentation of urban land cover
types from multispectral optical data. Thus, although extensive
image databases, such as ImageNet, were created
more than 10 years ago and form the backbone of many
modern ML developments in computer vision, there is
still no similar dataset or backbone network in remote
sensing. (Note that, as a prime example of an annotated
computer vision dataset, ImageNet contains more than
14 million images depicting objects from more than
20,000 categories.) This lack of generality renders the generation
of an ImageNet-like general EO dataset extremely
complicated and thus costly: instead of photographs
openly accessible on the Internet, many different-and
sometimes quite expensive-sensor data would have to
be acquired and, instead of " mechanical turks, " trained
EO experts would have to be hired to link these different
sensor data to the multitude of different domain- and
task-specific annotations (see " The Mechanical Turk " ).
Therefore, until now, the trend in ML applied to
EO data has been characterized by the generation of
The Mechanical Turk
The name mechanical turk comes from a fraudulent chess-playing machine
developed in the 18th century. Chess players were made to believe they played
against the machine, but were in fact competing against a person hidden inside
it. Today, the term mostly refers to Amazon Mechanical Turk (MTurk), a crowdsourcing
website run by Amazon. On MTurk, users can hire remotely located
crowdworkers to perform desired tasks. MTurk is frequently used to create
manual annotations for supervised machine learning tasks.
64
numerous remote sensing datasets, each consisting
of a particular combination of sensor modalities, applications,
and geographic locations. Yet, a review of
these developments is still missing in the literature.
The only articles that make a small step toward a general
review of benchmark datasets are [1], [2], [3], and
[4]. All of them provide some sort of review, however,
they are always limited to a very narrow aspect, e.g.,
object detection or scene classification. Furthermore,
their focus is on ML approaches and their corresponding
datasets, while the historical evolution of datasets
is neither discussed in detail, nor from a sensor- and
task-agnostic point of view.
As an extension of our 2021 IEEE International Symposium
on Geoscience and Remote Sensing contribution [5],
this article intends to close this gap by
◗ reviewing current developments in the creation of datasets
for DL applications in remote sensing and EO
◗ structuring existing datasets and discussing their properties
◗
providing a perspective on future requirements.
In this context, we additionally present
the Earth
Observation Database (EOD) [6], which is the result of
the effort and cooperation of voluntary scientists within
the IEEE Geoscience and Remote Sensing Society
(GRSS) Image Analysis and Data Fusion (IADF) Technical
Committee (TC). This database aims to function as
a centralized tool that organizes the meta information
about existing datasets in a community-driven manner.
EVOLUTION OF EO-ORIENTED ML DATASETS
HISTORICAL DEVELOPMENT
High-quality benchmark datasets have played an increasingly
important role in EO for quite some time and are one
of the driving factors for the recent success of DL approaches
to analyze remote sensing data. As such, they can be seen as
a tool complementary to methodological advancements to
push accuracy, robustness, and generalizability. This section
reviews and summarizes the historical development of EOoriented
ML datasets to provide insights into the evolution
of this " tool, " ranging from its historical beginnings to the
current state of the art.
The beginnings of ML applied to remote sensing focused
on specific applications. Datasets were mainly built by considering
a very localized study site, a few specific sensor modalities,
and a relatively small number of acquired samples.
Therefore, the first datasets were relatively small compared
to what is now considered a benchmarking dataset. Training,
validation, and testing samples were often taken from
the same image. Even with the use of sophisticated shallow
learning models but especially since the advent of DL, such
small datasets were no longer sufficient for proper training
and evaluation. The need for extended datasets has led to
the creation of larger datasets containing multiple images,
often acquired at different geographic locations.
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2023
IEEE Geoscience and Remote Sensing Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2023
