IEEE Geoscience and Remote Sensing Magazine - December 2022 - 237

different settings for the minimum cropping size. The results
show that the bigger the minimum size (less aggressive
cropping), the worse the performance is. This is in line
with the computer vision literature, which emphasizes the
importance of having stronger augmentations to make the
training task hard enough for the network.
REGIME OF LIMITED LABELS
Though Figure 24 proves the capability of self-supervised
pretrained models, the linear probing results are still below
the upper bound of fully supervised learning with full
labels. Thus, we wonder how these pretrained models behave
when reducing the number of labels and turning on
fine-tuning. We pretrain ResNet18 with MoCo-v2 for 100
epochs and evaluate on BigEarthNet. Figure 26 shows the
linear classification and fine-tuning performance when reducing
the number of labeled samples to 50%, 10%, 1%,
and 0.1%. The general trend of the Figure proves the potential
of self-supervised pretraining when we do not have
enough labels; linear classification with self-supervision
outperforms vanilla supervised learning when reducing
the number of available labels to 1%. The advantage is
more significant for fine-tuning, which outperforms supervised
learning on all scenarios and provides huge
improvements on a tiny number of labels. In general, the
fewer labels we use, the bigger the advantage of self-supervised
pretraining is.
TRANSFER LEARNING
In addition, we report transfer learning performance by
linear classification and fine-tuning on EuroSat with a
ResNet18 pretrained on BigEarthNet. As is shown in Figure
27, the performance plots are similar to Figure 26,
proving the pretrained models transfer the data representations
across datasets well. Promisingly, fine-tuning on
50% of labels outperforms supervised learning with full
labels. In fact, fine-tuning on 10% of labels already gives
a very close performance compared to full-label supervised
learning.
CHALLENGES AND FUTURE DIRECTIONS
SSL has been achieving great success in various vision tasks,
yet it is still a relatively new branch of technology in both
the computer vision and remote sensing communities. In
this section, we summarize the challenges of SSL in remote
sensing and discuss possible future directions.
MODEL COLLAPSE
Model collapse is one of the main challenges along with
the development of SSL, especially for modern contrastive
learning. Though it has been shown that contrastive SSL
has achieved great success in the field, the fundamental
theory behind those contrastive methods (especially those
using no negative samples) is still not well understood.
There are already some works, like SimSiam [84] and [249],
that try to investigate the underlying theory, yet they are not
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
enough. In fact, as has been mentioned in the " A Preliminary
Benchmark " section, SimSiam easily gets collapsed
in Earth observation data. Future works need to go deeper
into self-supervised representation learning and explore
the theoretical foundations behind model collapse in remote
sensing data.
PRETEXT TASKS AND DATA AUGMENTATIONS
Pretext tasks and data augmentations play a similar and
very important role in SSL as they correspond directly to
what invariance the networks need to learn and what information
is important for image understanding. There
have been some early studies on the evaluation of different
pretext tasks and data augmentations and possible improvements
[250], [251], yet they mainly focus on natural
45
50
55
60
65
70
75
80
85
90
MoCo-v2 SwAV
SimSiam
Barlow Twins
FIGURE 25. A study of data augmentation on BigEarthNet. The
baseline includes all augmentations " No_Crop " means all augmentations
except cropping, and " Crop_Only " means using only
cropping as the augmentation with minimum cropping size 0.2,
0.5, and 0.8 of the original size, respectively.
90
85
80
75
70
65
Linear
Fine-Tuned
Supervised
0.1 110
Percentage of Labeled Data
50
100
FIGURE 26. BigEarthNet linear classification and fine-tuning results
using different numbers of labels pretrained with MoCo-v2. Selfsupervision
outperforms supervision when reducing the number
of labels.
237
Mean Average Precision (%)
Baseline
No_Crop
Crop_Only (0.2)
Crop_Only (0.5)
Crop_Only (0.8)
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
