IEEE Geoscience and Remote Sensing Magazine - December 2022 - 219

Denoising AE is trained to enforce robustness against
noise in data. Its reconstruction loss gets modified to
xD (( )) ,
-+e < with e a noise modeling term.
VAE decouples the encoder and the decoder by encoding a
latent distribution. For each input sample ,xi
gle deterministic latent representation ,zi
i ,
<
Ex
instead of a sinthe
encoder E generates
two latent vectors i
n and v representing the mean and
variance of a latent Gaussian distribution. The decoder then
randomly samples a representation vector zi
the Gaussian distribution (, )( )/
reconstruct x ,i (see Figure 4).
N ii
nv +exp z
ii i
2
sampled from
6--nv2@ to
The required mathematical machinery to properly define
the VAE training loss builds on variational inference [91]. However,
in practice, it amounts to adding a loss term that forces the
latent distribution to a unit Gaussian, i.e., (, )( ,)
for all training samples .xi
N nv
ii N "
the KL divergence between the two distributions
(( ,) (, )) ().
KLNN01 2
1
nv <
22 122
=- vn v++ -
log
01
This can be realized by minimizing
(1)
Together with the reconstruction term, the total loss of the
VAE reads
LVAE=- mv nv+- ++ -1
xD Ex
(( )) () (2)
log
22 2
where m is a weighting parameter.
Most recently, masked AE (MAE) [58] raised great attention
in the computer vision community with a breakthrough
in autoencoding self-supervised pretraining of
ViTs. Inspired by denoising AE, MAE masks out random
patches of the input image, sends visible patches to the encoder,
and reconstructs the missing patches from the latent
representation and masked tokens. This work proves the
potential of Transformer-based AEs for self-supervised visual
representation learning.
AUTOENCODERS IN REMOTE SENSING
AEs have been widely used to learn representation from
various remote sensing data like multispectral images
[92]-[99], HSIs [100]-[107], and SAR images [108]-[111].
Lu et al. [92] proposed a combination of a shallowly
weighted deconvolution network with a spatial pyramid
model to learn multilayer feature maps and filters for input
images. In a subsequent step, these get classified by a support
vector machine for scene classification. Zhang et al.
[93] utilized a stacked denoising AE to learn image features
for the change detection of imagery with various spatial
resolutions. In HSI analysis, AEs are either exploited for
pretraining [101] or become ingredients in common downstream
tasks, such as HSI classification [100]; HSI denoising
[112]; HSI restoration [102], [113]; and HSI unmixing
[99], [103], [107].
GENERATIVE ADVERSARIAL NETWORKS
GANs [59], [114] proposed the adversarial training framework-a
strategy ruled by minimax optimization [115].
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
GAN training may be viewed as a two-player game; the
generator G generates fake samples
xzG=r
() from random
latent vector z, and the discriminator D aims to distinguish
xr
D x 01!
unified form of the GAN loss reads
() [( ())]
maxminLDxz1 DG=+ -
GAN
D G
loglog
from real data samples x.
When the discriminator's output is interpreted as probability
distribution, i.e., () [, ], a widely adopted,
(3)
where x represents a real sample and ()zG represents a fake
sample. During training, G is optimized to fool the discriminator
by maximizing (( ))zD G for any z, while D
is tuned to minimize ()xD r
for fake data xr and maximize
D ()x for real data x. Accordingly, the parameters of G and
D get simultaneously tuned to minimize and maximize
L ,GAN
respectively.
Contrary to AEs where zE (),x=
G .1To
approximate G 1zE
()x=
a GAN's latent representation
z is implicitly modeled through the inverse of the
generator,
and obtain z for a given
sample x, adversarial AEs (AAEs) [60] can be used. For real
samples x, G gets fed by
of a (trained) AE DE%
The discriminator contributes to the loss by distinguishing
fakes
xzG=r
() from real samples xE xG=u
(( )). Once ,G ,D
and DE% are simultaneously optimized, E serves to encode
data points x into learned representations z.
To make GAN's training scheme more symmetric, BiGAN
[61], [116] and ALI [117] have been proposed. As depicted
by Figure 5, in addition to generating fake data
xzG=r
tations zxE=r
distinguish tuples of fakes (, )xz
rr
from real ones, (x, z).
GENERATIVE ADVERSARIAL NETWORKS
IN REMOTE SENSING
While GAN-based methods in remote sensing are rare for
self-supervised pretraining, several works tend to integrate
such methods to target applications. Zhu et al. [118]
presented a first work to explore the potential of GAN for
HSI classification. Jin et al. [119] proposed using AAE for
unsupervised hyperspectral unmixing. Hughes et al. [108]
proposed a GAN-based framework to generate similar but
novel samples from a given image. Subsequently, these are
Feature
z
G
Data
G(z)
G(z), z
E(x), x
E Ex(x)
FIGURE 5. Bidirectional GAN (BiGAN) [61]. BiGAN includes an
encoder E that maps data x to latent representations z. The BiGAN
discriminator D jointly acts in data and latent space: x versus
G ()),z and E(x) versus z, respectively.
219
D
P(y)
()
through G , an encoder E generates latent space represen().
A discriminator D (, )xzD=
is trained to
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
