IEEE Geoscience and Remote Sensing Magazine - September 2019 - 37

calibration field, where coded markers are distributed in
three dimensions with known accurate positions [49]. This
method provides highly precise calibration parameters, but
it is high in cost, inconvenient, and unsuitable for frequent
recalibration in UAV-RS. An alternative, low-cost solution is
based on a 2D calibration pattern, e.g., a checkerboard [45],
a completely flat liquid crystal display-based method [46],
or an AprilTag-based method [47]. It has been demonstrated
that this solution can achieve an accuracy close to that of
a 3D physical calibration field. Different patterns are designed to improve the accuracy and ease of feature detection
and recognition under various conditions.
It is worth noting that reference object-based calibration usually requires preprepared calibration patterns and
extra manual operations that make it laborious and time
consuming. By contrast, self-calibration, which depends
on structural information detected in images without requiring special calibration objects, is more flexible and efficient. It has, therefore, become an area of active research
in recent years, especially for automatic rectification and
calibration of fisheye images.
Among these methods, geometric structures (e.g., conics, lines, and plumb lines) are first detected [43], [50],
[51]. If given at least three conics on a distorted image, the
camera's intrinsic parameters can be obtained from the decomposition of absolute conics. A fisheye image is generally rectified based on the assumption that a straight line
should maintain its line property even after the projection
of a fisheye lens. Several approaches have been proposed to
extract geometric structures, such as the extended Hough
transform [52] and multilabel energy optimization [53].
However, the effects of rectification are often limited by the
accuracy of the geometric structure detection.
More recently, deep convolutional neural network (CNN)based methods have been proposed, which try to learn
more representational visual features to rectify the distorted image [54]. The work in [55] proposed an end-to-end
deep CNN that learns semantic information and low-level
appearance features simultaneously to estimate the distortion parameters and correct the fisheye image. However,
this technique does not consider geometry characteristics,
which are strong constraints in rectifying distorted images. To deal with this issue, Xue et al. [56] designed a deep

(a)

(b)

network to exploit distorted lines as explicit geometry constraints to recover the distortion parameters of the fisheye
camera and rectify the distorted image.
Some fisheye image rectification examples based on
self-calibration are shown in Figure 8. The qualitative
evaluation of a fisheye data set is reported in Table 3. It can
be seen that deep CNN-based methods (e.g., [56]) achieve
excellent rectification performance for fisheye images.
However, some problems remain to be solved. The encoding of other geometries, such as arcs and plumb lines, into
CNNs is still a formidable issue. Designing robust geometric feature detection methods, especially in the case of
noise or low texture, is also an area requiring research. Another important challenge is to improve self-calibration
to achieve an accuracy comparable to reference objectbased approaches.
COMBINED FIELD OF VIEW
Because of the low flight altitude and narrow FOV of cameras on UAVs, UAV-RS often acquires images with a small
ground coverage area, resulting in increased image numbers, flight lines, flight cost, and data collection time [58].
One solution to these issues is the combined wide-angle
camera, which uses multiple synchronized cameras. The
images acquired from this multicamera combination system (i.e., an equivalent large-array camera) are rectified,
registered, and mosaicked to generate a larger virtual image, which can augment the coverage area [49]. In contrast
to narrow cameras, the combined wide-angle method can
increase acquisition efficiency and enlarge the base-height
ratio. It also benefits the image connection, especially in
some windy conditions. Another advantage is obtaining
multiview images by oblique acquisition, which can overcome photographic dead areas and sheltered targets. In
[59], the combined wide-angle camera is used for photogrammetric surveying and 3D building reconstruction. Figure 9 shows an example of a four-camera system.
The combined wide-angle camera has been well studied in the UAV-RS community. However, improving its
acquisition efficiency for larger-area mapping remains a
concern. In this regard, an emerging opportunity is multiUAV collaboration, which uses fleets of simultaneously
deployed, swarming UAVs to achieve a remote sensing

(c)

(d)

FIGURE 7. Examples of camera calibration: (a) a 3D physical calibration field, (b) checkerboard calibration [45], (c) the dual liquid crystal

display-based method (taken from Zhan [46]), and (d) the AprilTag-based technique [47].
SEPTEMBER 2019

IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE

IEEE Geoscience and Remote Sensing Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - September 2019