IEEE Geoscience and Remote Sensing Magazine - December 2022 - 217

LINK TO SEMISUPERVISED LEARNING
SSL algorithms may be considered as part of semisupervised
learning-a branch of machine learning concerned
with labeled and unlabeled data. Van Engelen and Hoos
[4] proposed a comprehensive taxonomy of semisupervised
classification algorithms based on conceptual and
methodological aspects for processing unlabeled data.
Within that scheme, SSL may get categorized as unsupervised
preprocessing; in the first step, unlabeled data get
transformed to extract feature representations; labeled
data are then utilized to adapt the model to specific tasks.
Within the remote sensing community, semisupervised
learning has been long studied and employed in applications
in, e.g., hyperspectral image (HSI) recognition and
processing [19]-[28]; multispectral
image segmentation
[29]-[35]; and synthetic-aperture radar (SAR)-optical data
fusion [36].
LINK TO UNSUPERVISED LEARNING
SSL is also a discipline in the realm of unsupervised
learning-the general set of machine learning methods
that are independent of human annotation. However, in
various communities, the difference between self-supervised
and unsupervised learning is very ambiguous. In
this article, we separate them in terms of methodology.
Traditional unsupervised algorithms tend to utilize the
statistics of the input data and generate groups by dimensionality
reduction [37], [38] or clustering [39]. On the
other hand, SSL is a recent term that refers to approaches
in which a model (e.g., a neural network) is trained to
learn good data representations using supervision signals
that are automatically generated from the data themselves.
In remote sensing, unsupervised learning has also
been actively employed in various applications, e.g., in
scene classification [38], [40], [41]; semantic segmentation
[42]-[44]; change detection [45], [46]; and multisensor
data analysis [47], [48].
PERFORMANCE EVALUATION
IN SELF-SUPERVISED LEARNING
To evaluate the performance of SSL methods, downstream
tasks are commonly defined to transfer the pretrained
model f to specific applications, such as scene
classification, semantic segmentation, and object detection.
The performance of transfer learning on these
high-level vision tasks estimates the generalizability of
the model f. In practice, three common procedures are
used for quantitative evaluation.
◗ Linear probing (or linear classification): This refers to fixing
the parameters of the learned model f (frozen encoder)
and training a linear classifier g on top of the generated
representations. This approach measures how linearly
separable the embeddings produced by the pretrained
model f are.
◗ K-nearest neighbors: K-nearest neighbors (KNN) refers to
applying weighted voting of the KNN of the input test
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
image x in the feature space. This approach is nonparametric
and can be used for classification tasks.
◗ Fine-tuning: This refers to training a model on the downstream
task by using the parameters of the pretrained
model f as an initialization. It is the most general procedure
since it is not limited only to classification.
In addition to quantitative evaluation, qualitative visualization
methods can also be used to derive insights on
the features learned during the self-supervised training of
the encoder f. Among these, two popular techniques are
kernel visualization [49]-[51] for CNN and feature map
visualization [49], [52], [53] for both CNN and ViT. Kernel
visualization plots kernels of the first convolutional
layer of f and compares them with kernels from the corresponding
fully supervised training. The similarity of
kernels learned by supervised and self-supervised training
sheds light on the efficacy of self-supervision. Along
these lines, feature map visualization displays hidden
layer feature maps of f to analyze the spatial attention of
f on the input x. Once again, the feature maps are compared
with their counterparts of supervised training for
reference.
GENERATIVE, PREDICTIVE, AND CONTRASTIVE:
A TAXONOMY OF SELF-SUPERVISED LEARNING
In this section, we review works in SSL following an extended
taxonomy: generative, contrastive, and predictive methods,
as displayed in Figure 3. Generative methods learn to
reconstruct or generate input data; predictive methods learn
to predict self-generated labels; and contrastive methods
learn to maximize the similarity between semantically identical
inputs. While some existing categorizations solely classify
self-supervised methods into generative and contrastive
Unlabeled Data {x}
x1
Self-Supervision
xn
Labeled Data {x′, y′ }
x′1
Downstream
Task
x′n
FIGURE 2. The general pipeline of SSL. The visual representation
is learned through self-supervision that comes from the unlabeled
data. The learned parameters serve as a pretrained model and are
transferred to supervised downstream tasks for fine-tuning.
217
...
...
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
