IEEE Geoscience and Remote Sensing Magazine - June 2022 - 142

features [37], [38] and GPU-based hardware acceleration
[39], have also been exploited in the last decades. All of
these algorithms have been widely used in RS applications.
The comparative evaluation of these algorithms can be
found in the references of [31] and [40]-[42].
LEARNED METHODS
In contrast to the mentioned hand-crafted features, recent
years have seen the rapid development of learned local features
based on deep learning. In general, learned features
are extracted from feature maps that are generated by using
convolutional neural network (CNN) models. Unlike the
elegantly designed hand-crafted methods that are directly
executed on images, CNN models are first trained by using
training samples and are capable of predicting local features
with the aid of learned parameters [43].
For feature detection and description, existing CNN
learning networks can be divided into two main groups,
i.e., separate and joint detector and descriptor networks. For
the former one, CNN models are only utilized to learn feature
descriptors from input image patches that are cropped
around key points. As one of the outstanding networks,
[44] designed the L2-Net with an all-convolution structure
and proposed a progressive sampling strategy and
log-likelihood lost function for network training. Since its
prominent performance on descriptor learning, L2-Net has
become a baseline network, and it is extended subsequently
in HardNet [45] using a triple margin loss, GeoDesc [46]
with geometry constraints from multiview reconstruction,
and ContextDesc [47] encoding spatial relationships from
key-point locations.
For joint detector and descriptor networks, key-point
detectors and descriptors are learned simultaneously from
end-to-end CNN networks. The inputs are usually raw images
instead of image patches in the separate detector and
descriptor networks. As an earlier attempt, [48] proposed
the learned invariant feature transform to simulate the individual
step in the classical feature extraction pipeline,
including feature detection, orientation estimation, and
feature description.
However, it is not practical to train feature-detection
networks with human-labeled key points. To facilitate the
issue, [49] designed the SuperPoint, which achieves the selfsupervised
training of feature detectors using self-labeled
key points extracted from a synthetic data set. In contrast to
the mentioned methods that use the detect-then-describe
approach, [50] proposed the D2-Net to address the featurematching
problem under extreme appearance changes,
which postpones the detection stage after feature description
and achieves feature detection using high-level information.
D2-Net can be an efficient alternative for processing
historical UAV images with large illumination changes.
As demonstrated in recent research [43], the separate
detector and descriptor learning networks achieve the best
performance in SfM-based image orientation, as they combine
the high precision of hand-crafted feature detectors
142
and increased discriminative power of learned feature descriptors.
Also, they can be an alternative in the existing
pipeline of local feature-based matching and have been
exploited in 3D reconstruction applications [46], [47]. Recent
work demonstrates that learned features using separate
detector and descriptor networks can achieve comparable
or better performance in SfM and multiview stereo (MVS)based
3D reconstruction for UAV images.
All of this newly reported research confirms the potential
of these learned features as alternatives to classical 3D
reconstruction workflows. However, the main drawback
of learned detectors is the relative lower location accuracy
[51]. For more details about learning-based local features,
these works can be found in [43], [52], and [53].
NEW CHALLENGES
Despite the explosive research, new challenges are arising
in the feature detection and description of UAV images.
This can be seen from two aspects. On the one hand, local
support regions (LSRs) of corresponding key points are
very different due to serious occlusions and perspective
deformations caused by low flight altitudes and oblique
imaging angles; on the other hand, to adapt to the limited
memory capacity, the widely used downsampling strategy
for high-resolution images would degenerate the precision
of feature locations and reduce the number of extracted
features. To address these new challenges, some work has
been documented:
◗ Different LSRs: Affine-invariant regions [31] have been
exploited to extract LSRs with the sacrifices of decreasing
feature numbers and increasing time costs. On the
other hand, using auxiliary data from onboard GNSS/inertial
navigation system (INS) sensors, global geometric
rectification can be applied for the relief of perspective
deformations, which has been used in feature matching
of oblique aerial [55] and UAV images [54], as shown in
Figure 9. Figure 10(a) and (b) presents matching results
without and with global geometric verification.
◗ Large image dimensions: Image downsampling as well as
divide and conquer are widely used strategies to address
the out-of-memory problem, especially for the GPUbased
SIFT. The image downsampling strategy, however,
would cause the loss of location accuracy and a reduction
in the number of features. Thus, for processing images
with large dimensions, such as aerial and UAV images,
the divide-and-conquer strategy is utilized in the fields
of photogrammetry and RS [54], [56]. For image pairs
without other auxiliary data, the homography transformation
estimated from downsampled images is used to
establish corresponding blocks; with the assistance of a
digital surface model (DSM) and the intrinsic and extrinsic
parameters of cameras, corresponding blocks can be
determined through object space projection. The influence
of the divide-and-conquer strategy on image matching
is illustrated by Figure 10(c) and (d), which increases
match numbers and improves their distributions.
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE JUNE 2022
IEEE Geoscience and Remote Sensing Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - June 2022
