IEEE Geoscience and Remote Sensing Magazine - December 2021 - 130

originated in computer vision and has a long history [154].
In recent years, it has gradually entered use in remote sensing
image applications, such as image fusion [155], [156],
LC classification [157], [158], and segmentation [159]. The
framework is data driven and can generate image features
by learning from many training data sets with a specified
principle [158]. Therefore, it is suitable for remote sensing
image registration.
Some studies have focused on feature matching for this
purpose [158], [160]. Most utilize a Siamese network consisting
of two parts to train a deep NN (DNN) [161]-[164].
One part extracts features from image patch pairs by training
a Siamese, pseudo-Siamese, or improved Siamese network
[165]; the other part measures the similarity between these
features for image matching. In [164], the DNN inspired the
construction of a deep learning framework for remote sensing
image registration. In addition, generative adversarial networks
(GANs) are applied to image matching and registration
[166], [167]. These approaches first translate an image into
another one by training the GANs, enabling two images to
have similar intensities and feature information [166], [168].
Feature extraction and matching are subsequently performed
between two artificially generated images, effectively improving
the performance of image matching. For the deficiencies
of specified-scale NNs, multitask learning is introduced to
improve the registration precision [169]. Wang et al. break
through the limitations of the traditional deep learning approach,
which extracts image features in one network and
matches them with the other NN. They design an end-toend
network using forward propagation and backward feedback
to learn the mapping functions of the patches and their
matching labels for remote sensing image registration [164].
Recently, Li et al. paired image blocks from sensed and reference
images and directly learned the displacement parameters
of four corners of the sensed block relative to the reference image
on a deep learning regression network, which differs from
the traditional deep learning method [170].
Deep learning has advantages over the traditional registration
approach. It is completely data driven and has strong
flexibility, enabling it to theoretically fit any complex mapping
function, whereas the traditional registration method
can deal only with fixed pattern registration. Moreover, deep
learning extracts abstract and high-level semantic information.
Compared with low-level gray and gradient data, deep
semantic information is more consistent with the way humans
understand images. Therefore, deep learning methods
can extract robust features. However, deep learning has challenges.
It highly depends on image samples; when there is a
lack of data or the data quality is poor, deep learning methods
have difficulty ensuring the effectiveness of the registration
results. Although remote sensing images are now easy to
acquire, the lack of manual annotation and standard data is
still very serious. Deep learning, in essence, learns the statistical
characteristics of a large number of similar images, but
its input-output process is a complex, nonlinear mapping
without clear physical significance. Additionally, deep learning
requires high computing power and has major hardware
requirements, limiting its applicability.
In short, remote sensing image registration based on deep
learning is still in its infancy, and its registration framework is
not mature. However, many studies have demonstrated that
deep learning methods can achieve or even surpass the optimal
level of traditional registration approaches in terms of accuracy
and efficiency. We predict that deep learning-based methods
will become important solutions to the problem of real-time
and high-precision remote sensing image registration.
REGISTRATION BASED ON
THE COMBINATION METHOD
As mentioned, feature- and intensity-based approaches have
their own advantages. Different feature extractors also have
various precisions. To integrate these strengths as fully as possible,
combination techniques have been developed. Typically,
popular combinations consist of two aspects, namely,
feature- and area-based approaches; however, some integrate
two geometric feature-based approaches, such as the SIFT and
Harris detectors.
FIGURE 12. Feature matching examples.
130
COMBINATIONS OF FEATURE- AND
AREA-BASED ALGORITHMS
Feature-based approaches are typically suitable for images
with more significant structural data than intensity information.
However, they are restricted by the distribution and
accuracy of the features. On the other hand, area-based approaches
are appropriate for images with more distinctive
intensity information; however, they require the intensity
information of the reference and sensed images to be correlated.
Thus, the two methods have complementary pros and
cons. To further improve registration accuracy and robustness,
some studies focus on a combination of geometric featureand
area-based techniques [171]. Huang et al. [172] proposed
a hybrid approach to aligning images by intensities within
a scale-invariant feature region. Elsewhere, a wavelet-based
feature extraction technique and an area-based method with
NCC were combined to reduce the local distortion caused by
terrain relief [173]. In a wavelet-based hierarchical pyramid
framework, Mekky et al. [174] proposed a hybrid approach
using MI and the SIFT; employing the rough registration parameters
of the area-based approach for MI, the number of
false alarms obtained by the SIFT was reduced. In addition,
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE DECEMBER 2021
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
