IEEE Geoscience and Remote Sensing Magazine - September 2023 - 74

diverse datasets becomes apparent, in particular, covering
underdeveloped countries as well.
SOURCE OF THE REFERENCE DATA
Another important aspect is the source of the reference data.
Although most of the scientific papers that introduce a new
benchmark dataset are very detailed regarding source and
properties of the provided image data, they often contain
only sparse information about the exact process of how the
reference data were obtained. However, knowledge about
the source of the reference data as well as about measures
for quality assessment and quality control are essential to
judge the potential of the benchmark and how to interpret
obtained results. For example, achieving high " accuracy "
on a dataset that contains a high number of annotation errors
in its reference data only means that the model is reproducing
the same bias as the method used to annotate the
data but is not actually producing accurate results. Furthermore,
information about the source of the reference data is
not only scarce but also very heterogeneous. Examples include
manual annotation for semantic segmentation [e.g.,
OpenEarthMap (Table 2; 253)] or object detection [e.g.,
BIRDSAI (Table 2; 127)], DroneCrowd (Table 2; 138), the
use of keywords in image search engines for classification
[e.g., AIDER (Table 2; 34)], leveraging existing resources
[e.g., BigEarthNet (Table 2; 40)] uses the CORINE land use
land cover (LULC) map, MapInWild (Table 2; 264) uses
the World Database of Protected Areas, AI-TOD (Table 2;
162) uses other existing datasets, ship detection datasets
usually employ AIS data, OpenSentinelMap (Table 2; 282)
uses OSM, automatic reference methods (e.g., HSI Oil Spill
108
11
14
16
20
23
27
31
36
42
48
54
61
69
78
87
97
1
2
4
5
7
9
CD
Superresolution/Pansharpening
Object Detection
Classification
Semantic Segmentation
Change Detection and Classification
Object Detection and Classification
Object Detection and Semantic Segmentation
Classification and Semantic Segmentation
Others
(Table 2; 211) and Sen12MS-CR-TS (Table 2; 98), or a mixture
of these [e.g., RSOC (Table 2; 117)] uses existing datasets
(DOTA, Table 2; 109) as well as manual annotations for
new images.
The quality of class definitions varies among different
datasets, often depending on whether a dataset is designed
for a specific real-world application (in which class definitions
are driven by the application context), or whether it
is just an example dataset used to train and evaluate a MLbased
approach (in which case, class definitions are more
arbitrary). If human interaction is involved (e.g., via manual
annotations), annotation protocols or precise class definitions
are often not shared (if they even exist). With the
evolution of datasets, the quality of the meta-information
about the reference data needs to increase, together with
the quality and quantity of the image data.
APPLICATIONS AND TASKS
Finally, we analyze a last aspect concerning the distribution
of some characteristics of the datasets, namely, the
number of classes and images. In Figure 7(a), we show the
evolution of the number of classes per publication year.
In Figure 7(b), we plot the number of classes against the
number of images in the dataset. In both plots, the radius
indicates the size of the image (width or height), while the
color indicates the type of task. We can see that there is no
clear trend or correlation between the year of publication and
the number of classes in a dataset. Instead, we find that more
recently published datasets increase the variety in the number
of classes, again highlighting an increased interest in using
benchmarking datasets in a wider range of applications.
108
11
14
16
20
23
27
31
36
42
48
54
61
69
78
87
97
1
2
4
5
7
9
Publication Year
(a)
Number of Images
(b)
FIGURE 7. The number of classes provided by the reference data of a given dataset not only varies for different tasks (e.g., object detection
is dominated by datasets with only a single class) but also with (a) publication year and (b) the number of images of a dataset.
74
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2023
Number of Classes
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
1
6
28
357
109
1,000
2,000
6,000
14,000
29,000
59,000
112,000
206,000
365,000
627,000
1,048,000
1,708,000
2,719,000
4,242,000
6,492,000
IEEE Geoscience and Remote Sensing Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2023
