IEEE Geoscience and Remote Sensing Magazine - December 2021 - 79

huge difference between VHR remote sensing images and
data from other fields (e.g., natural RGB images) in terms
of the image modality, spectral bands, spatial resolution,
viewing angle, and so on, large amounts of publicly available
multitemporal VHR remote sensing data are required
to construct a robust VHR deep change detector. On the
other hand, semisupervised deep learning methods, with
the consideration of unlabeled samples [120], can relieve
the burdensome labeling process, although the effects of
unlabeled samples as well as the complexity of the semisupervised
model should be further investigated.
With regard to the fusion strategy, according to how
bi-temporal images are dealt with, deep learning-based change
detectors can be classified as early fusion and late fusion. Early
fusion methods concatenate multitemporal images as a whole
input into a deep network [110]. Early fusion is able to capture
the hierarchical difference representation, i.e., from low-level
grayscale differences in shallow layers to high-level semantic
changes in deep layers, while grayscale differences that are
not relevant to semantic changes, e.g., spatial misalignment
and the internal variability of objects, may propagate
to deeper layers and therefore lead to false alarms [113].
In contrast to early fusion, late fusion methods separately
learn monotemporal features and concatenate them later
as an input to the change detection layers [121]. This kind
of network architecture may lead to insufficient learning,
e.g., during network training. Gradients in high layers are
difficult to flow backward to lower ones [122] and hence affect
the change detection performance. Thus, as an attempt
in [113], early and late fusion networks were combined to
complement one another.
As for network models, AE [123], [124], deep belief networks
[125], CNNs [110], [112], [113], [115], [120], [126],
recurrent neural networks (RNNs) [127]-[129], generative
adversarial networks (GANs) [130], [131], and graph neural
networks [132] have been adopted for end-to-end change
detection. The CNN is one of the most widely used methods,
and mainstream CNN architectures, such as AlexNet
[133], VGGNet [134], GoogleNet [135], ResNet [136], and
DenseNet [137] as well as their variants, have been considered
[138]. RNNs with modules, such long short-term
memory and gated recurrent units as well as their variants,
are also widely employed to model the phenological process
of multitemporal VHR images, due to the superiority of
recurrent layers in processing sequential data and modeling
time-series dependence. In addition, the U-net and its variants,
which are composed of an encoder to hierarchically
extract semantic information and a counterpart decoder to
delineate spatial details, can be viewed as AE architectures
for VHR change detection. They receive much attention due
to their ability to maintain change object spatial details.
Recently, some studies proposed hybrid models, such
as those in [111] and [127]. For instance, as illustrated in
Figure 9, a CNN and an RNN are combined in one endto-end
network to extract joint spectral-spatial-temporal
features [111]. In [139], difference-based methods using
DECEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
edge-based level set evolution (ELSE), region-based level
set evolution (RLSE), MRFs, and fully convolutional networks
(FCNs) as well as postclassification-based methods
with SVMs, CNNs, GANs, Siamese convolutional networks
(SCNs), and end-to-end GAN-based Siamese frameworks
(GSFs) are compared for landslide detection (Figure 10).
Since observing landslides separately from unchanged
and other changed regions is required, this kind of change
detection is challenging. As can be seen, the four difference-based
methods lead to more false alarms. As for the
five postclassification methods, deep learning techniques
generally outperformed SVMs, due to their explorative capabilities
in representing related changes and suppressing
irrelevant variations.
TABLE 4. A SUMMARY OF DEEP LEARNING-BASED
CHANGE DETECTORS.
CRITERIA CATEGORY DESCRIPTION
Learning
strategy
Supervised Based on a large number of
labeled samples
Transfer
learning
Metalearning
Semisupervised
Fusion
strategy
Fine-tunes
pretrained
models that are designed for
different but related tasks
Learns from little labeled
data and learns how to learn
Joint use of labeled and
unlabeled data
Early fusion Uses concatenated multitemporal
images as input
Late fusion Learns monotemporal
features separately and
then concatenates them
as a whole input
Network
model
CNN
Recurrent
neural
network
AE
Deep belief
network
Graph neural
network
Generative
adversarial
network
Processing
unit
Patch
Pixel
Object
Stacked convolutional,
pooling, and fully
connected layers
Models with a recurrent
hidden state, e.g., gated
recurrent units and long
short-term memory
Reconstructs the input
with an encoder-decoder
structure
Composed of layer-wise restricted
Boltzmann machine
Learns graph structure, e.g.,
relationships between features
of pixels/objects
Generator and discriminator
that are adversarially trained
REFERENCES
[110]-[116],
[121], [124]-
[130], [139], [140]
[117], [131]
[118]
[120], [132]
[110], [114], [115],
[125]-[129],
[131], [132], [139]
[111], [112], [116],
[117], [121],
[130], [140]
[110], [112]-
[116], [120], [126]
[127]-[129]
[123], [124]
[125]
[132]
[130], [131], [139]
Assigns a label to each patch [111], [115]-[117],
[120], [121],
[128]-[130]
Predicts change labels for
each pixel
Incorporation of segments/
superpixels
[110], [113],
[114], [126],
[131], [139]
[124], [125],
[127], [132], [140]
79
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
