IEEE Geoscience and Remote Sensing Magazine - December 2022 - 233

this characteristic can raise issues with data augmentation.
Specifically, data augmentation is very important,
especially for contrastive methods to help the network
learn semantic invariance, yet a careless design of data
augmentation might change certain physical properties,
which are indeed important to recognize the objects in a
remote sensing image.
Third, remote sensing images usually contain many
more objects than natural images (e.g., compared to an
image from ImageNet, which generally contains one main
object, a remote sensing image usually includes multiple
repeated objects). This difference influences both pretext
tasks and data augmentation. For example, jigsaw puzzles
can be expected to be difficult for representation learning
on low-resolution satellite images, and rotation can affect
the shadows of buildings, which are important for the
model to learn relative heights. On the other hand, when
image-level representation is important for natural images,
patch-level and pixel-level representation can be equally
important for remote sensing images (e.g., considering the
spatial relationship of neighboring and distant patches to
sample positive/negative pairs [177], [196] as shown in Figures
13 and 20).
Last but not least, the time variable is becoming increasingly
important despite the modality. This raises the interest
of getting inspiration from video SSL, yet time stamps
in remote sensing images are quite different from frames
in videos as well as the special objectives (e.g., change detection
in remote sensing). Therefore, how to make the
best of temporal information is a common question to be
answered in all modalities. Early progress is shown in Figure
21, where seasonal information is used for data augmentation
[201].
Apart from these important common characteristics,
later we discuss different modalities of remote sensing
data. A list of recent self- supervised works classified with
regard to (w.r.t.) modality is shown in Table 2.
MULTISPECTRAL IMAGES
Multispectral images are captured within specific wavelengths
ranging across the electromagnetic spectrum,
which allows for the extraction of information covering
far beyond the perception range of human eyes. Compared
to hyperspectral
imaging, multispectral
sensing multispectral image analysis, but there is some
special attention to pay:
1) The common characteristics of remote sensing data for
all modalities need to be considered, as has been discussed
previously.
2) Spectral contexts of predictive methods (e.g., colorization)
now have possibly more channel information that
can be used to design the pretext tasks.
3) Color-related data augmentations (e.g., the color distortions
shown in Figure 12), which are very important for
contrastive SSL, need to be carefully modified to fit into
multiple channels.
HYPERSPECTRAL IMAGES
Hyperspectral imaging also captures images across the electromagnetic
spectrum but lights in a far bigger number
(up to hundreds) of spectral bands than multispectral
imaging with a tradeoff on the spatial resolution. This
very high spectral resolution enables us to identify the
E
T0
T1
T2
E
E
E
h0
h1
h2
Invariant
Invariant
+
+
Invariant
FIGURE 21. Seasonal Contrast (SeCo) [201]. Three embedding
subspaces are trained to be invariant to temporal (seasonal),
synthetic, and both temporal and synthetic transforms T1, T2, and
T0, respectively.
Publications w.r.t.
Different Modalities
Publications w.r.t.
Different SSL Types
imaging
measures light in a small number (typically 3-15) of
spectral bands with relatively low spectral resolution but
high spatial resolution. Multispectral images are the most
commonly used remote sensing data as they are relatively
easy to acquire and are close to natural images, which enable
easier human perception and convenient technology
transfer from the computer vision community. Due to
the popularity and the similarity of multispectral images
to natural images, a lot of self-supervised methods have
been directly transferred to the remote sensing field (see
Figure 22). Generally, all categories of self-supervised
methods in computer vision can be extended to remote
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
Multispectral
Hyperspectral
SAR
Generative
Predictive
Contrastive
Multimodal
(a)
(b)
FIGURE 22. Statistics of recent publications related to SSL in
remote sensing. (a) Modality. (b) SSL type. w.r.t.: with regard to.
233
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
