IEEE Geoscience and Remote Sensing Magazine - September 2023 - 98

SOFTWARE AND DATA SETS
YI WANG , NASSIM AIT ALI BRAHAM, ZHITONG XIONG ,
CHENYING LIU , CONRAD M. ALBRECHT, AND XIAO XIANG ZHU
SSL4EO-S12
A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation
S
elf-supervised pretraining bears the potential to
generate expressive representations from large-scale
Earth observation (EO) data without human annotation.
However, most existing pretraining in the field is
based on ImageNet or medium-sized, labeled remote
sensing (RS) datasets. In this article, we share an unlabeled
dataset Self-Supervised Learning for Earth Observation-Sentinel-1/2
(SSL4EO-S12) to assemble a large-scale,
global, multimodal, and multiseasonal corpus of satellite
imagery. We demonstrate SSL4EO-S12 to succeed
in self-supervised pretraining for a set of representative
methods: momentum contrast (MoCo), self-distillation
with no labels (DINO), masked autoencoders (MAE),
and data2vec, and multiple downstream applications,
including scene classification, semantic segmentation,
and change detection. Our benchmark results prove the
effectiveness of SSL4EO-S12 compared to existing datasets.
The dataset, related source code, and pretrained
models are available at https://github.com/zhu-xlab/
SSL4EO-S12.
INTRODUCTION
Self-supervised learning (SSL) has attracted wide attention
in the RS community with the ability to learn generic
representations from unlabeled data. Numerous
studies in the literature have proven the potential of SSL
in EO beyond natural images [1]. Despite the focus SSL
for EO receives, only limited effort is dedicated to providing
large-scale datasets and benchmarks for pretraining.
On the one hand, relying on computer vision datasets
like ImageNet [2] is not a preferred option due to the
domain gap. On the other hand, while RS datasets like
SEN12MS [3] or seasonal contrast (SeCo) [4] exist, they
are limited by geospatial overlap, sparse geographical
distribution, or lack diversity in seasonal or multimodal
Digital Object Identifier 10.1109/MGRS.2023.3281651
Date of current version: 19 September 2023
98
information. Therefore, big EO-specific datasets for unsupervised
pretraining are necessary to be developed.
In this work, we introduce a large-scale, globally
distributed, multitemporal and multisensor dataset,
SSL4EO-S12. The dataset samples 251,079 locations
around the globe, each providing Sentinel-2 level-1C
(L1C), Sentinel-2 level-2A (L2A), and Sentinel-1 ground
range detected (GRD) images with four snapshots from
different seasons (in total: 3 million ,,2 640 2 640
-m #
-m
patches). Additionally, we guarantee optimal geospatial
coverage by avoiding the overlap of the randomly
sampled locations. This renders SSL4EO-S12 the largest
and most generic multispectral/synthetic aperture radar
(SAR) dataset in the RS literature [5].
We demonstrate the potential of the SSL4EO-S12 dataset
through a series of extensive experiments. Specifically,
we evaluate four representative SSL algorithms-namely:
MoCo [6], DINO [7], MAE [8], and data2vec [9]-on
three different downstream tasks: scene classification, semantic
segmentation, and change detection. Our results
indicate that pretraining on SSL4EO-S12 improves the
downstream performance compared to existing datasets.
Moreover, our ablation studies prove the benefits of RSspecific
data augmentations, including multisensor, multitemporal,
and atmospheric correction.
RELATED WORK
SELF-SUPERVISED LEARNING
Over the past years, SSL has reached important milestones
in computer vision, especially through contrastive
methods with joint-embedding architectures. These
methods get trained to promote similarity between
augmented views of the same input, thereby enforcing
invariance to data augmentation. Several families of
such methods emerge: 1) contrasting negative samples
for which the representations are encouraged to be dissimilar
[6]; 2) knowledge distillation between an asymmetric
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE SEPTEMBER 2023

https://orcid.org/0000-0002-3096-6610 https://orcid.org/0000-0002-3953-585X https://orcid.org/0000-0001-9172-3586 https://orcid.org/0000-0001-5530-3613 https://github.com/zhu-xlab/SSL4EO-S12 https://github.com/zhu-xlab/SSL4EO-S12

IEEE Geoscience and Remote Sensing Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2023