IEEE Geoscience and Remote Sensing Magazine - December 2022 - 238

images and evaluate on image-level classification tasks. As
has been proved in the " A Preliminary Benchmark " section,
the findings from common computer vision benchmarks
may not completely hold for multispectral imagery,
not to mention other modalities. In the " From ImageNet
to Remote Sensing Imagery: SSL on Geospatial Earth Observation
Data " section, we discussed the characteristics
of different remote sensing data, yet still, experiments
are needed. Therefore, in general, more studies on pretext
tasks and data augmentations are required to better
understand which ones are useful for different types of
remote sensing data.
PRETRAINING DATASETS
Most of the existing remote sensing datasets are intended
for supervised learning. Though they can also be used for
self-supervised pretraining by discarding the labels, the
amount of data is limited, and the data are usually biased
toward the tasks of the datasets. Therefore, the necessity
of self-supervised pretraining on large-scale uncurated
datasets needs to be studied. There have been some early
contributions following this thread. Leenstra et al. [186]
proposed a Sentinel-2 multitemporal cities pairs dataset
for change detection; Manas et al. [201] proposed a largescale
self-supervised Sentinel-2 dataset; and Heidler et al.
[202] proposed an audiovisual dataset for multimodal SSL.
However, there is still much space to explore and contribute
(e.g., dataset size, image sampling strategies, and the availability
of more modalities).
MULTIMODAL/TEMPORAL
SELF-SUPERVISED LEARNING
As one of the most important characteristics of remote sensing
data, multimodality is a significant aspect to be explored
in self-supervised representation learning [151], [214],
[220]. On the other hand, multitemporal image analysis is
raising more interest because of the increasing frequency of
data acquisition and transferring. Without the need for any
human annotation, self-supervised representation learning
has a big advantage to performing big data analysis, which is
usually a challenge for multimodal and multitemporal data.
However, adding modalities and time stamps will also bring
complexity to the model to be trained. Thus, how to balance
different modalities or time stamps such that the model can
learn good representations from both modalities is a challenging
task to be explored.
COMPUTING-EFFICIENT
SELF-SUPERVISED LEARNING
Self-supervised pretraining usually requires a large amount
of computing resources because of big pretraining data;
various data augmentations; the requirement of large batch
sizes and more training epochs than supervised learning;
and so forth. Few efforts have been made toward reducing
computational cost, which is, however, an important factor
for practical usage. Therefore, efficient data compression
[252], data loading, model design [58], and hardware acceleration
are necessary to be explored.
NETWORK BACKBONES
Most of existing self-supervised methods utilize ResNet
[17] as their model backbones, while there has been a
hot trend toward ViTs [18]. Since ViT has shown promising
results in SSL [52], [77], [85], and recent advances
are exploring the link between large-scale pretraining
for both language and vision [58], [86], it deserves more
exploration in remote sensing images as well. In addition,
the specific encoder architecture itself is also important
to study as most existing methods tend to keep
it unchanged with a focus on the conceptual design of
self-supervision.
TASK-ORIENTED WEAKLY SUPERVISED LEARNING
Last but not least, in parallel to the progress of task-agnos100
90
95
85
80
75
70
65
50
Linear
Fine-Tuned
Supervised
110
Percentage
of Labeled Data
100
FIGURE 27. EuroSAT linear classification and fine-tuning results
pretrained with MoCo-v2 on BigEarthNet. Self-supervision with
50% labels outperforms supervised learning with full labels.
238
CONCLUSION
This article provides a systematic introduction and literature
review of SSL for the remote sensing community. We
summarize three main categories of self-supervised methods,
introduce representative works, and link them from
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2022
tic SSL, there's also the need to integrate " few labels " or
" weak labels " for better task-dependent weakly supervised
learning in practice. In other words, not only can SSL provide
a pretrained model for downstream tasks, but there's
also the possibility of bringing representation learning
online. For example, both data representations and the
supervised task can be learned together, where the selfsupervised
branch helps the model capture more useful
information. This scenario is especially important when
moving to extreme applications that do not have a common
large dataset for pretraining. In addition, this can also
be a way to mitigate the generally high computational cost
of training SSL algorithms.
Overall Accuracy (%)
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
