IEEE Geoscience and Remote Sensing Magazine - September 2021 - 175

problem, a common approach is to exploit DL models with
proven architectures (such as ResNet [7] or VGG [8]), which
are pretrained on publicly available general-purpose data
sets in the computer vision (CV) community (e.g., ImageNet
[9]). The existing model is then fine-tuned on a small set
of RS images annotated with multilabels to calibrate the
final layers. This strategy is also known as a transfer learning
strategy. There are several versions of the aforementioned
models that have been pretrained on large-scale data sets in
CV. However, we argue that this is not a proper approach in
RS because of the differences in image characteristics in CV
and RS. For example, Sentinel-2 multispectral images have
13 spectral bands associated with varying and lower spatial
resolutions compared to the CV images. In addition, the semantic
content present in CV and RS images is significantly
different, and thus the respective semantic classes differ
from each other.
To address this issue, we have recently introduced BigEarthNet
[10] as a large-scale, single-modal benchmark
archive for RS image search, retrieval, and classification.
BigEarthNet contains 590,326 Sentinel-2 image patches
annotated with multilabels provided by the CLC map of
2018 (CLC 2018) [11]. The CLC nomenclature includes land
cover and land use classes grouped in a three-level hierarchy,
and for the BigEarthNet image patches, the most thematically
detailed level-3 class nomenclature is considered.
However, there are some CLC classes that are difficult to
identify by only exploiting (single-date) Sentinel-2 images
because 1) land use concepts associated with some classes
(e.g., Dump sites, Sport and leisure facilities) may not be visible
from space or fully recognizable with the spatial resolution
of Sentinel-2 images, and 2) RS time series, which
BigEarthNet does not include, may be required to describe
and discriminate some classes (e.g., Nonirrigated arable land
and Permanently irrigated land). In addition, BigEarthNet is
not suitable for multimodal learning-based algorithm development
and validation purposes since it only contains
Sentinel-2 image patches.
To overcome these issues, in this article we introduce
BigEarthNet-MM, which contains 590,326 pairs of Sentinel-2
and Sentinel-1 image patches. We also introduce an
alternative nomenclature for images in BigEarthNet-MM
as an evolution of the original CLC labels. Figure 1 shows
an example of the BigEarthNet-MM image pairs and their
multilabels from the new nomenclature.
A DESCRIPTION OF BIGEARTHNET-MM
BigEarthNet-MM contains 590,326 pairs of Sentinel-1 and
Sentinel-2 image patches acquired over 10 different European
countries (Austria, Belgium, Finland, Ireland, Kosovo,
Lithuania, Luxembourg, Portugal, Serbia, and Switzerland).
Sentinel-2 patches of BigEarthNet-MM are taken from
the original BigEarthNet [10]. To construct these patches,
125 tiles associated with less than 1% of cloud cover and
acquired between June 2017 and May 2018 were considered.
All tiles were atmospherically corrected by employing
SEPTEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
Sentinel-2's level-2A product generation and formatting tool
(sen2cor) provided by the European Space Agency (ESA)
because of its proven success in the literature. After the
atmospheric correction, the 10th band of each image patch
is no longer available as it is the cirrus band (which is omitted
in the level-2A output
for its lack of surface information).
Then the tiles were
divided into 590,326 nonoverlapping
image patches,
each of which is a section of
1) 120 × 120 pixels for 10-m
bands, 2) 60 × 60 pixels for
20-m bands, and 3) 20 × 20
pixels for 60-m bands.
One important goal during
the tile-selection process
was to represent all chosen
geographic locations with
images acquired in different
seasons. However, because
of the restrictions of finding
tiles with a low cloud cover
percentage in the relatively narrow time period, this has not
been possible at each considered location. Accordingly, the
following respective numbers of patches for autumn, winter,
spring, and summer have been considered: 143,557; 72,877;
175,937; and 126,913. For a quality check of patches, visual
inspection was also employed, which led to the identification
of 70,987 Sentinel-2 image patches that are fully covered
by seasonal snow, cloud, and cloud shadow. (The lists
are available at http://bigearth.net/#downloads.)
To construct the patches of BigEarthNet-MM, 325 SenWE
HAVE MADE ALL THE
DATA AND THE DL MODELS
PUBLICLY AVAILABLE AT
HTTPS://BIGEARTH.NET,
OFFERING AN IMPORTANT
RESOURCE TO SUPPORT
STUDIES ON MULTIMODAL
IMAGE SCENE
CLASSIFICATION AND
RETRIEVAL PROBLEMS
IN RS.
tinel-1 Ground Range Detected products acquired between
June 2017 and May 2018 that jointly cover the area of all
original 125 Sentinel-2 tiles with close temporal proximity
were selected and processed. The selected scenes provide
dual-polarized information channels (VV and VH) and are
based on the interferometric wide swath mode, which is the
main acquisition mode over land.
All scenes were preprocessed by using the Sentinel-1
toolbox and the graph processing framework of the ESA's
Sentinel Application Platform. This includes the application
of precise orbit files, border and thermal noise removal,
radiometric calibration, and geometric correction (i.e.,
Range Doppler Terrain Correction). Depending on the spatial
extent of the scene, either the Shuttle Radar Topography
Mapping (SRTM) 30 (for scenes below 60° latitude) or the
Advanced Spaceborne Thermal Emission and Reflection
Radiometer Digital Evaluation Model (for scenes above 60°
latitude, where no SRTM 30 exists) was employed in the
geometric correction to project images from slant range to
ground range. Finally, the backscatter coefficient was converted
to a decibel scale.
It is worth noting that, since the selection of the speckle
filter is considered to be application dependent, to preserve
175
http://bigearth.net/ http://bigearth.net/#downloads
IEEE Geoscience and Remote Sensing Magazine - September 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2021
