IEEE Geoscience and Remote Sensing Magazine - December 2022 - 223

with G generating the self-supervised data representation,
as displayed in the " Generative Methods " section.
The method yields improved, robust differentiation for
change detection. Yuan and Lin [152] proposed a Transformer-based
self-supervised method for satellite time
series classification. By predicting randomly contaminated
observations given an entire time series of a pixel,
the model is trained to leverage the inherent temporal
structure of satellite time series to learn general-purpose
spectral-temporal representations. The work was further
improved in [153], where the network is asked to regress
the central pixels of the masked patches for patch-based
representation learning.
OTHER SEMANTIC CONTEXTS
Apart from the aforementioned three types of contexts,
there are also other semantic contexts that can be seen
using the integrated information of the previously mentioned
contexts. Noroozi et al. [67] proposed an artificial
supervision signal based on counting visual primitives
from an input image and the sum of primitives from
cropped tiles. Jenni and Favaro [68] proposed an SSL
method based on spotting and predicting artifacts in an
adversarial manner. To generate images with artifacts,
the authors pretrain a high-capacity AE and then implement
a damage-and-repair strategy. A discriminator is
finally trained to distinguish artifact images and predict
what entries in the feature were dropped when damaging
and repairing.
Multisensor or multimodal SSL gathers another set of
semantic contexts. Owens et al. [69] proposed to predict
a statistical summary of the sound associated with a video
frame. By defining explicit sound categories, the authors
formulate this visual recognition problem as a classification
task. Ren and Lee [154] trained an encoder-decoder
network to predict the depth map, surface normal map,
and instance contour map from a synthetic image, and a
discriminator network is trained to distinguish between
features from the synthetic image and the real image.
With various pretext tasks proven to be useful for selfsupervised
representation learning, there's also a trend of
gathering different pretext tasks together. A direct example
can be seen in [155], where the authors investigated the
combination of four self-supervised tasks: relative position
[62]; colorization [65]; exemplar [156] (creating a pseudoclass
from the augmented view of a single image and predicting
the class; this is rather a contrastive method, which
will be discussed in the next section); and motion segmentation
[157] (the network learns to classify which pixels
of a single frame will move in subsequent frames). It was
shown that combining self-supervised tasks will in general
improve the performance and lead to faster training.
OTHER SEMANTIC CONTEXTS IN REMOTE SENSING
Due to the difference between remote sensing data and common
images studied in the computer vision community,
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
there's also a large potential for designing remote sensing-specific
pretext tasks. Hermann et al. [158] proposed
to predict different views (pose, projection, and
depth) as well as reconstruction of the input for monocular
depth estimation. Gao et al. [159] proposed to
train the model to realize envelope data extrapolation
from a frequency band to their adjacent low-frequency
band. He et al. [160] proposed to reconstruct the corrupted
seismic data with consecutively missing traces in
which the pseudolabels are automatically created from
the uncorrupted parts of the observed data. Ayush et al.
[161] designed a pretext task based on predicting the
geolocation of the input image. Li et al. [162] made use
of GlobeLand30 [163], a global land cover product that
divides Earth into different areas and includes 10 different
land cover types. By aligning the geolocation of
the input image with the land cover map from GlobeLand30,
the land cover class of the input image is then
used as a pretext task for SSL. However, it also has to
be noted that pretexts like geolocation itself are usually
too simple to learn a good representation, and thus, they
are often used as auxiliary tasks for SSL (as in the two
examples mentioned earlier). In general, how to design
or integrate a suitable pretext task is still an important
research question to be explored.
CONTRASTIVE METHODS
The performance of predictive SSL depends largely on a
good pretext task, which is often very difficult to design and
may even lead to pretext-specific representation, decreasing
the network's generalizability. To tackle this problem, contrastive
methods come into play, giving the network more
freedom to learn high-level representations that do not rely
on a single pretext task. As the name implies, contrastive
methods train a model by contrasting semantically identical
inputs (e.g., two augmented views of the same image) and
pushing them to be close by in the representation space. (In
some literature, the term contrastive is used only to denote
methods that include negative samples. Yet we follow the
more common terminology in which contrastive methods
also include the recent works without negative samples.)
Therefore, by design, contrastive methods usually follow
a common Siamese-like architecture design. A side-by-side
comparison with generative and predictive methods is provided
in Figure 8.
However, enforcing only similarity between pairs
of input can easily lead to a trivial solution. Indeed,
a constant mapping would be a valid solution to the
problem since every pair of input would have identical
representations, and thus, a maximum similarity. This
phenomenon is often referred to as model collapse, and
many solutions have been proposed in the SSL literature
to mitigate it. In fact, depending on how collapsing is
handled, one can form a subtaxonomy of contrastive SSL:
negative sampling, clustering, knowledge distillation, and
redundancy reduction.
223
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
