IEEE Geoscience and Remote Sensing Magazine - December 2022 - 230

enforcing point-level consistency across views. They also
propose a background-swap augmentation to focus on
the foreground.
L CC 2///
2
BT _ -+m
44i 4444
^h
1
ii
12 3 >
i ji
!
invariance term
redundancy reduction term
REDUNDANCY REDUCTION
The idea of redundancy reduction for SSL comes from neuroscience.
Indeed, Barlow et al. [225] state that the goal of
sensory processing is to record highly redundant sensory
inputs into a factorial code (a code with statistically independent
components). Inspired by this principle, Zbontar
et al. [87] proposed Barlow Twins, a method that uses
redundancy reduction as a way to avoid a trivial solution
in contrastive learning without explicit (e.g., MoCo [75]
and SimCLR [76]) or implicit (e.g., DeepCluster [51] and
SwAV [81]) negative samples. With the main network architecture
similar to previous contrastive learning, Barlow
Twins' objective function measures the cross-correlation
matrix between the embeddings of two identical networks
fed with distorted versions of a batch of samples and tries
to make this matrix close to the identity (Figure 16). This
causes the embedding vectors of distorted versions of a
sample to be similar while minimizing the redundancy between
the components of these vectors. The loss function
of Barlow Twins is
T1
x
T2
x2
Eθ
P
z2
Feature
Dimension
FIGURE 16. Barlow Twins [87]. The objective function measures
the cross-correlation matrix between the embeddings of two
identical networks fed with distorted versions of a batch of samples
and tries to make this matrix close to the identity. This causes the
embedding vectors of distorted versions of a sample to be similar
while minimizing the redundancy between the components of
these vectors. Cross-Corr.: cross-correlation.
x1
Eθ
Projector
P
z1
Empirical
Cross-Corr.
Target
Cross-Corr.
REDUNDANCY REDUCTION IN REMOTE SENSING
Since redundancy reduction-based algorithms (Barlow
Twins [87] and VICReg [88]) are relatively new, there
are only a few works using these methods in the remote
sensing community. Marsocci and Scardapane [226] proposed
using Barlow Twins in a continual learning setting
to pretrain a model on a set of heterogeneous datasets
while avoiding catastrophic forgetting. Their approach
is evaluated on segmentation problems. Ait Ali Braham
et al. [227] used Barlow Twins for few-shot HSI classification
in a two-stage approach. They propose two pairsampling
strategies for pixel- and patch-level classification
using spatial information from the scene to generate
where m is a tradeoff parameter, and C is the cross-correlation
matrix computed between the outputs of the two identical
networks along the batch dimension
Cij _
//
/
^^
bi
A
b
hh
b
2
bj
B
zz
zz
b
,,
,
bi
A
,
bj
B
2
(9)
where b indexes batch samples; i and j index the vector dimension
of the networks' outputs; and A and B represent
two different views.
Bardes et al. [88] borrowed the decorrelation mechanism
from Barlow Twins, proposing a new redundancy reduction-based
self-supervised method VICReg (variance-
invariance-covariance regularization). VICReg follows
common contrastive learning architecture (see Figure 17)
and avoids the trivial solution by introducing variance
along the batch dimension (a hinge loss that constrains
the standard deviation); invariance across different views
(mean-squared Euclidean distance); and covariance along
the feature dimension (penalizing the off-diagonal coefficients
of the covariance matrix of the embeddings).
ij
(8)
Contrastive
Similarity
Sinkhorn
Encoder
Encoder
Encoder
Encoder
Predictor
Student
Teacher
Encoder
Encoder
Similarity
Redundancy
Image
(a)
Image
(b)
Image
(c)
Image
(d)
FIGURE 17. A comparison of contrastive self-supervised methods. (a) Contrasting negative samples. (b) Clustering. (c) Distillation. (d)
Redundancy reduction.
230
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2022
Grad
Grad
Grad
Grad
Grad
Grad
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
