IEEE Geoscience and Remote Sensing Magazine - December 2022 - 234

materials contained in the pixel via spectroscopic analysis,
yet on the other hand, requires special care about
band-wise analysis. In fact, with a basic objective of dimensionality
reduction, PCA/ICA/clustering and manifold
learning-based methods have been widely studied in
the field before the era of neural networks. Along these
lines, deep learning-based self-supervised methods have
been focusing on AEs and 3D convolution to compress
spectral representations. Because of the large number of
spectral channels, the network usually takes a single pixel
or a small patch as input. In both cases, spatial contexts
for pretext tasks or data augmentations tend to consider
intrasample instead of inner-sample information (e.g.,
neighboring patches or pixels can be seen as positive samples
in contrastive negative sampling). Meanwhile, spectral
contexts have a large potential to be explored because
of the high spectral dimensionality of hyperspectral data.
For example, two spectrally divided parts of an HSI can be
two augmented views.
SAR IMAGES
SAR is a form of radar that is used to create 2D images or 3D
reconstructions of objects. Due to the imaging principle,
SAR images are substantially different from multispectral
and HSIs in characteristics like dynamic range, speckle statistics,
imaging geometry, and complex domain information.
While SSL with SAR images is also a new research direction
and has not been widely explored, there's one fact
that makes it a valuable topic: the prevailing lack of ground
truth for regression-type tasks. Though simulators can be
used to provide training data for supervised learning, this
bears the risk that the networks will learn models that are
far too simplified. Therefore, it would be very beneficial if
we can use self-supervised techniques to learn representations
from real SAR data. In addition, the general selfsupervised
schemes can also be easily transferred to SAR
images with specific considerations. For example, the data
augmentation of color distortion is not suitable for SAR
data; the speckle noise of SAR images has a similar effect as
Gaussian blurring; and the ascending and descending order
may serve as a natural augmentation strategy.
MULTISENSOR FUSION
As a result of the multimodality of remote sensing data,
multisensor fusion is always an important task for various
objectives, which can benefit from multimodal SSL.
There are generally two ways of combining multimodal
inputs:
1) Similar to various input channels, different modalities
can be seen as augmented views of semantically the
same input, which can be used for designing pretext
tasks or specific data augmentation in contrastive SSL.
2) Two encoders are used separately to encode features
from two different modalities and then concatenated
together for a joint representation learning.
Regarding remote sensing data, a common multimodality
is from optical and SAR images, leading to the important
topic of SAR-optical data fusion, which has been shown by
some recent works that benefit from self-supervised representation
learning [214] (see Figure 23) and [220]. In addition,
the fusion of other modalities, like audio and social
media data, is also raising increasing interest [202].
APPLICATIONS
Though the general goal of SSL is to perform task-independent
representation learning, which can benefit various
downstream tasks, it has to be noted that different
self-supervision can have different influences on different
tasks. Therefore, it is necessary to preconsider the choice of
self-supervised methods when we are targeting a specific
application in the end. In fact, so far, a large number
of self-supervised works in the remote sensing field are
based on a specific application. In general, the various applications
can be separated into three categories: imagelevel
tasks, pixel-level tasks, and patch-level tasks. A list
of representative recent self-supervised works classified
w.r.t. applications is shown in Table 2. We refer the interested
reader to the numerous surveys on machine
learning and deep learning in remote
sensing [8], [239]-[244] for
more details about common downstream
tasks and available datasets.
Image Space
Image Pairs
ResUNet Projection Alignment
FIGURE 23. Shift invariance [214]. The two inputs have an offset but keep an overlap. A shift
transformation is included in the one branch for aligning representations between two branches,
which are optimized by a contrastive loss.
234
IMAGE-LEVEL TASKS
Image-level tasks correspond to the applications
that expect the recognition
of the whole image or image patches,
focusing more on global knowledge.
The most common image-level task is
scene classification for multispectral
[92], [130], [133], [213]; SAR [235];
and other possible modalities. Scene
classification is directly related to
natural image classification and is
usually the default downstream task
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2022

IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022