IEEE Geoscience and Remote Sensing Magazine - September 2023 - 100

downloaded four images drawn from four annual seasons
to capture seasonal variation. We searched for Sentinel-2
tiles with a cloud coverage lower than 10%. We also filtered
out most overlapping patches with an efficient grid search
strategy. In total, we obtained about 1 million S1-GRD/
S2-L1C/S2-L2A image triplets.
DATA IDENTIFICATION
The collection of SSL4EO-S12 differs from SeCo mainly by
introducing overlap filtering and multiple sensors (in italics
below). The workflow is shown as follows:
1) Uniformly sample one city from top-10,000 populated
cities.
2) Sample one location from a Gaussian distribution with
a standard deviation of 50 km around the city center.
3) Check if a ,, image patch centered around
26 mm40- # 2 640that
location has significant overlap with previous patches. If
not, continue to step 4, otherwise return to step 1.
4) For a 30-d interval around four reference dates (20
March, 21 June, 22 September, 21 December) in 2021
(additionally look for 2020 as a buffer), check if there
exist Sentinel-2 tiles with less than 10% of cloud coverage
(for both L1C and L2A) and corresponding Sentinel-1
GRD tiles.
5) If there exist valid Sentinel-1/2 tiles close to all the four
dates, process and download them into curated image
patches, otherwise return to step 1.
OVERLAP FILTERING
A simple way to check significant overlap between two
patches is to calculate the distance between the two centers.
If the distance is smaller than three-quarters of the
width of a patch, there is a nonnegligible overlap (>25%).
Naively, we needed to execute this computation for every
new patch relative to all existing patches. However,
this becomes inefficient when the number of patches
grows large, 250,000+ for us. Therefore, we employed a
grid search strategy to perform efficient overlap filtering.
Instead of calculating the distance to all previous
patches, we distributed the patch center coordinates into
360 180#
geographical longitude-latitude, 11#cc grids.
For each new patch, we converted the center coordinates
into integer grid coordinates. Subsequently, we searched
for existing patches within this grid cell and exclusively
calculated distances to those local patches. Assuming potential
overlap of sampled patches from distinct grid cells
is statistically negligible, we significantly reduced computing
time compared to a global overlap search. Indeed,
for SSL4EO-S12 we recorded an overlap for approximately
3% tiles of densely populated Tokyo, Japan, 1.5% in
Chicago, IL, USA, and below 1% for locations such as Beijing,
China, Munich, Germany, Kampala, Uganda, and
Brasilia, Brazil.
DATA CHARACTERISTICS AND VOLUME
The presented SSL4EO-S12 dataset contains 251,079
globally distributed Sentinel-1 dual-pol SAR, Sentinel-2
top-of-atmosphere multispectral, and Sentinel-2 surface
reflectance multispectral triplets over four seasonal timestamps.
As of summer 2022, SSL4EO-S12 constitutes the
biggest geospatial-temporal, multimodal dataset in terms
of medium-resolution PolSAR and multispectral imagery,
serving more than 3 million images. The total data volume
equates to an uncompressed size of ,[ B
() ]. .
251 079 42 4## $ +
13 12 2 264 37TB
+ $# 2 .
B
Figure 2 depicts the geospatial distribution of the
SSL4EO-S12 dataset, highlighting the dense coverage
across the globe. Figure 3 depicts the effect of overlap filtering
around the Tokyo area.
EXPERIMENTAL SETUP
We evaluated the SSL4EO-S12 dataset by self-supervised
pretraining and transfer learning on RS downstream tasks.
Specific implementation details and additional results are
provided in the supplemental material (available at https://
doi.org/10.1109/MGRS.2023.3281651).
SELF-SUPERVISED PRETRAINING
We performed pretraining using four representative
SSL methods: MoCo-v2/v3 [15], [16], DINO [7], MAE
[8], and data2vec [9]. We pretrained ResNet [17] backbones
with MoCo(-v2) and DINO, and Vision Transformer
(ViT) [18] backbones for all four SSL methods
listed above. Unless explicitly noted, Sentinel-2 L1C
was used for pretraining. To utilize multitemporal information,
we used RandomSeasonContrast as a data
augmentation strategy. For MoCo and DINO, the input
views were randomly picked from two seasons. For
MAE and data2vec, one random season was assigned
for each patch.
Pretraining one ResNet/ViT model for 100 epochs takes
7-25 h on four NVIDIA A100 GPUs, as shown in Table 2.
FIGURE 2. Geographical distribution of SSL4EO-S12 dataset.
100
TRANSFER LEARNING
The pretrained models were transferred to various downstream
tasks. For scene classification, we evaluated EuroSAT
[19] (single-label land cover classification), BigEarthNet [13]
(multilabel land cover classification), and So2Sat-LCZ42
[20] (local climate zone classification, culture-10 version).
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2023
https://doi.org/10.1109/MGRS.2023.3281651 https://doi.org/10.1109/MGRS.2023.3281651
IEEE Geoscience and Remote Sensing Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2023
