IEEE Geoscience and Remote Sensing Magazine - December 2022 - 236

SimSiam due to collapse, which will be discussed later) as
it is designed for natural RGB images. Four NVIDIA A100
GPUs are used for the experiments, on which the batch size
is in total 256. Links to the four methods' official sites and
details of all the hyperparameters we use can be found in our
code repository. In downstream tasks, we transfer 100-epoch
pretrained encoders for MoCo, SwAV, and Barlow Twins and
10-epoch pretrained encoders for SimSiam (the model collapses
afterward). In addition, we report the performance of
fully supervised learning and training with randomly initialized
encoders for more thorough comparison.
BENCHMARK RESULTS
GENERAL COMPARISON OF SSL METHODS
The general benchmark results with linear probing evaluation
are shown in Figure 24(a). A first conclusion is that with
pure simple transfer from natural images to satellite images,
all the self-supervised methods provide meaningful
10
20
30
40
50
60
70
80
90
BigEarthNet So2Sat-LCZ42
Random MoCo-v2
SEN12MS
SwAV
SimSiam Barlow Twins Supervised
(a)
-0.955
-0.965
-0.975
-0.985
-0.995
0102030405060708090 100
Epochs
(b)
FIGURE 24. (a) A preliminary benchmark of four popular contrastive
self-supervised methods (each representing one category
in the " Contrastive Methods " section) on Sentinel-2 images of
BigEarthNet [13], SEN12MS [14], and So2Sat-LCZ42 [15] (culture-10
version) datasets. We use ResNet-18 as encoder backbones, follow
the official settings of the models for self-supervised pretraining,
and evaluate the performance by training a linear classifier on
frozen features. (b) The " collapsing " training curve of SimSiam. The
loss goes down quickly in the beginning 10 epochs and becomes
very unstable in the following epochs.
236
DATA AUGMENTATION
In this section, we provide an extended study on the impact
of data augmentations that have been proved to be very important
for natural images [76]. Focusing on BigEarthNet,
we do self-supervised pretraining considering a rich set of
augmentations: " RandomResizedCrop " ; " RandomColorJitter "
(a simplified version that changes only contrast and
brightness); " RandomGrayscale " ; " RandomGaussianBlur " ;
" RandomHorizontalFlip " ; and " RandomChannelDrop "
(replacing one or several channels by zero). As can be seen
in Figure 25, the results show both common and different
trends compared to the computer vision community. When
dropping only the " RandomResizedCrop " transform, SwAV
and Barlow Twins collapse immediately, and MoCo-v2 and
SimSiam see a significant performance drop. All methods
confirm the importance of cropping, which is in line with
natural images.
Surprisingly, almost no visible performance drop is observed
when using only cropping as the data augmentation
for all four methods. This is a significant difference compared
to natural images, where other augmentations (especially
color distortion) bear big importance as well. Therefore,
apart from cropping, which is a key augmentation, further
study is required on the design of other augmentations to
work well on remote sensing multispectral images. In addition,
to further analyze the impact of cropping, we explore
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2022
representations for all datasets (compared to random initialization).
With frozen features that come from label-independent
pretraining, we can train much fewer parameters
(e.g., a full ResNet-18 has 10 million parameters, while the
last linear layer contains only 500,000) with comparable
performance toward fully supervised training.
The second observation is that MoCo-v2 [78], which
represents contrastive negative sampling, generally outperforms
the other methods across datasets. This proves the
robustness and transferability of using negative samples for
SSL in remote sensing, which is also consistent with the fact
that most of the recent self-supervised works in the remote
sensing community prefer MoCo [75] and MoCo-v2 [78] as
a baseline structure.
A third observation is that, compared to the other
methods, the training of SimSiam [84] is very unstable.
As can be seen from the training curves in Figure 24(b),
the training loss of SimSiam drops very quickly and significantly
in the beginning epochs and starts vibrating
in the following epochs. This can be attributed to SimSiam's
simplified design (as has been described in the
" Contrastive Methods " section), which makes the similarity
matching task easy to solve for the models. When
utilizing the checkpoints of the later epochs to serve as
pretrained weights, the model fails to beat even random
initialization. That is to say, SimSiam gets collapsed after
10 epochs. Thus, in real applications, it is better to add a
momentum encoder or other tricks for better performance
and more stable training.
Loss
Classification Score (%)
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
