IEEE Geoscience and Remote Sensing Magazine - December 2021 - 145

Before discussing deep learning algorithms, we would
like to stress that the importance of high-quality benchmark
data sets in deep learning research cannot be overstated.
Especially in supervised learning, the knowledge
that can be gained by a model is bounded by the information
present in the training data set. For example, the
Modified National Institute of Standards and Technology
[25] data set played a key role in LeCun's seminal paper
about CNNs and gradient-based learning [26]. Similarly,
there would be no AlexNet [27], the network that kickstarted
the current deep learning renaissance, without
the ImageNet [28] data set, which contains more than
14 million images and 22,000 classes. ImageNet has been
such an important part of deep learning research that,
more than 10 years after it was published, it is still used
as a standard benchmark to evaluate the performance of
CNNs for image classification.
DEEP LEARNING MODELS
The main principle of deep learning models is to encode
input data into effective feature representations for target
tasks. To exemplify how a deep learning framework works,
we take the autoencoder as an example: it first maps input
data to a latent representation via a trainable nonlinear
mapping and then reconstructs inputs through reverse
mapping. The reconstruction error is usually defined as
the Euclidian distance between inputs and reconstructed
inputs. Parameters of autoencoders are optimized by gradient
descent-based optimizers, such as stochastic gradient
descent, root mean square propagation [29], and Adam
[30], during the backpropagation step.
CNNs
With the success of AlexNet in the 2012 ImageNet Large
Scale Visual Recognition Challenge (ILSVRC), where the
network scored a top-five test error of 15.3%, compared to
the second-best test error of 26.2%, CNNs have attracted
worldwide attention and are now used for many image understanding
tasks, such as image classification, object detection,
and semantic segmentation. AlexNet consists of five
convolutional layers, three maximum pooling layers, and
three fully connected layers. One of the key AlexNet innovations
was the use of graphics processing units (GPUs), which
made it possible to train such large networks with huge
data sets without using supercomputers. In just two years,
the Visual Geometry Group Network [2] overtook AlexNet
in performance by achieving a 6.8%, top-five test error at
ILSVRC-2014; the main difference was that it used only
3 × 3-sized convolutional kernels, which enabled it to have
more channels and, in turn, capture more diverse features.
The residual neural network (ResNet) [31], U-Net [32],
and DenseNet [33] were the next major CNN architectures.
Their main feature concerned the idea of connecting not
only neighboring layers but any two layers in the network
by using skip connections. This helped reduce information
loss across networks, mitigated the problem of vanishing
DECEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
gradients, and facilitated the design of deeper networks.
U-Net is one of the most commonly used image segmentation
networks. It has an autoencoder-based architecture
that uses skip connections to concatenate features from the
first layer to the last, the second to the second last, and so
on; this way, it can get fine-grained information from the
initial layers to the end layers. U-Net was initially proposed
for medical image segmentation, where data labeling is a
big problem. The authors employed heavy data augmentation
techniques on input data, making it possible to learn
from only a few hundred annotated samples.
In ResNet, skip connections were incorporated within
individual blocks, not across the whole network. Since its
initial proposal, ResNet has undergone many architectural
tweaks, and, even after five years, its variants are always
among the top scorers on ImageNet. In DenseNet, all the
layers were attached to all preceding layers, reducing the
size of the network, albeit at the cost of memory usage.
For a more detailed explanation of different CNN models,
interested readers are referred to [34]. These CNN models
have also proved their worth in SAR processing tasks; e.g.,
see [35]-[37]. For more examples and details of CNNs in
SAR, see the " Recent Advances in Deep Learning Applied
to SAR " section.
RECURRENT NEURAL NETWORKS
Besides CNNs, recurrent neural networks (RNNs) [38] are
a major class of deep networks. Their main building blocks
are recurrent units, which take the current input and output
of the previous state as input. They provide state-of-the-art
results for processing data of variable lengths, including text
and time-series information. Their weights can be replaced
with convolutional kernels for visual processing tasks, such
as image captioning and predicting future frames/points in
visual time series data. Long short-term memory (LSTM)
[39] is one of the most popular RNN architectures: its
cells can store values from any past instances and are not
severely affected by the problem of gradient diminishing.
As with any other time-series data from deep learning tool
kits, RNNs are natural choices to process SAR time-series
information; e.g., see [40].
GENERATIVE ADVERSARIAL NETWORKS
Proposed by Ian Goodfellow et al. [41], generative adversarial
networks (GANs) are among the most popular and exciting
inventions in the field of deep learning. Based on gametheoretic
principles, they consist of two networks called a
generator and a discriminator. The generator's objective is to
learn a latent space through which it can create samples
from the same distribution as the training data, while the
discriminator tries to learn to distinguish whether a sample
is from the generator or the training data. This very simple
mechanism is responsible for most cutting-edge algorithms
for various applications, e.g., generating artificial photorealistic
images and videos, superresolution, and text-toimage
synthesis. For example, in the SAR domain, GANs
145
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
