IEEE Geoscience and Remote Sensing Magazine - September 2023 - 104

downstream tasks. However, these product levels with
or without atmospheric correction can also be considered
natural data augmentation for SSL. Accordingly, we
conducted an ablation study on a 50,000 SSL4EO-S12
subset utilizing Sentinel-2 L1C, L2A, or both (L1C+L2A).
Table 10 summarizes our findings: 1) models pretrained
on the same product level as the downstream task have a
slight edge (~ 1%) over models trained on the other product
level, and 2) pretraining on both modalities generates
TABLE 10. LINEAR PROBING RESULTS OF DIFFERENT
PRODUCT LEVELS OF SENTINEL-2.
PRETRAIN PRODUCT
L1C
L2A
L1C+L2A
BE-10% (L2A)
74.0
75.1
78.0
Bold values indicate best per column performance.
EUROSAT (L1C)
93.1
92.0
93.8
a notable improvement of up to 4% compared to pretraining
on a single modality.
IMPACT OF PRETRAINING SCALE
An aspect relevant to large-scale data mining in EO is scaling
of results with training data volume: why don't we add
more images to SSL4EO-S12? One reason concerns computational
costs. We believe the current dataset (1 million
patches for each Sentinel product) is comparable to the scale
of ImageNet, and can serve as a good baseline in RS for
further development. Moreover, as observed by [25], saturating
downstream performance kicks in beyond 500,000
pretraining images on ImageNet, with 250,000 images
yielding acceptable results with as little as 1-2% accuracy
loss. We observe such a trend in our dataset, too. As demonstrated
by Table 11, we pretrained on various amounts of
data to report linear probing results for BigEarthNet-10%.
While 50% (500,000) or less pretraining data yields significant
performance drops, there's fewer diminishing
gaps from 75% (750,000) on. Note, this saturation effect
depends also on the model size.
TABLE 11. LINEAR PROBING RESULTS ON BIGEARTHNET-10%
FOR VARIOUS SENTINEL-2 L1C PRETRAINING DATA SIZES.
PRETRAIN DATA 100,000 250,000 500,000 750,000 1 MILLION
Accuracy (%)
64
73
78
81
82
REPRESENTATION VISUALIZATION
We qualitatively evaluated the data representations learned
from self-supervised pretraining by visualizing the latent
distributions with t-distributed stochastic neighbor embedding
(t-SNE) (Figure 5). We pretrained a ResNet50
with MoCo-v2 on SSL4EO-S12, and transferred the frozen
encoder to EuroSAT to calculate one 128-d representation
vector for each image. We then visualized all the vectors
with t-SNE, and compared the distribution with a randomly
initialized encoder.
(a)
CONCLUSION
In this work, we present SSL4EO-S12, a large-scale multimodal,
multitemporal unlabeled dataset for SSL in EO.
An extensive benchmark on various SSL methods and
RS applications proves the promising benefits of the proposed
dataset.
SSL4EO-S12 has some limitations: 1) there's little coverage
of polar regions, 2) geographical bias exists due to cloud
filtering, 3) it is not strictly free of geospatial overlap, and
4) medium-resolution radar and multispectral images are a
limited subset of EO data. Despite these limitations, we believe
SSL4EO-S12 renders a valuable basis to advance selfsupervised
pretraining and large-scale data mining in RS.
(b)
FIGURE 5. t-SNE visualization of EuroSAT image representations.
One color represents one class. (a) Random-encoded features;
(b) SSL-encoded features. SSL-encoded features are well clustered
even without label information.
104
ACKNOWLEDGMENT
This work is jointly supported by the Helmholtz Association
through the Framework of Helmholtz Artificial Intelligence
(Grant ZT-I-PF-5-01)-Local Unit " Munich Unit @Aeronautics,
Space and Transport (MASTr) " and Helmholtz Excellent
Professorship " Data Science in Earth Observation-Big
Data Fusion for Urban Research " (Grant W2-W3-100); by the
German Federal Ministry of Education and Research in the
framework of the International Future Artificial Intelligence
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2023
IEEE Geoscience and Remote Sensing Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2023
