IEEE Geoscience and Remote Sensing Magazine - December 2021 - 17

scenes and more than one million object instances with
HBB annotations. Compared with images in existing HRRS
data sets, xView images are high resolution, multispectral,
and labeled with a greater variety of objects. The images collected
from WorldView-3 have a resolution of up to 0.3 m.
The variations of scale, color, shape, and texture make the
data set more challenging to the remote sensing community.
HRRSD
The HRRSD data set is a large-scale benchmark with 21,758
RBG images extracted from Google Earth and has spatial
resolutions ranging from 0.15 to 1.2 m [42], [43]. There are
13 categories of objects, which allows this to be considered
an extended version of the NWPU VHR-10 data set with additional
classes, such as crossroads, parking lots, and T junctions.
This data set is class balanced, and each category has
3,700-5,000 instances.
DIOR
The DIOR data set is a recently released aerial DOTA [19]. It
contains 23,463 images with 800 × 800 pixels and 192,472
instances labeled with HBBs. The images were collected
from Google Earth and have resolutions ranging from 0.5
to 30 m. This data set has sufficient variations of scale,
weather, seasons, imaging conditions, and quality as well
as high interclass similarity and intraclass diversity. It is
also one of the larger-scale data sets, with massive images
and object instances.
COMPARISON
As shown in Figure 6, early HRRS data sets, such as SZTAKIIRIA
[36] and NWPU VHR10 [37], contained a small number
of categories and instances for the detection of large or
easily recognized objects. After several years, scholars have
forged ahead to introduce massive numbers of instances
and many categories: multisensor data, complex context,
and low-quality images to create large-scale challenging
data sets, such as xView [41], DIOR [19], HRRSD [42], [43],
and DOTA [18], which are becoming more and more in
line with the conditions of actual applications. These four
satellite data sets contain more than 13 object categories
and more than 50,000 object instances, with resolutions
ranging from 0.3 to 30 m, all available for the development
detectors that adapt to large-scale object detection. Some
representative small, weak examples from the aforementioned
data sets are collected in Figure 7(a). It can be seen
that the objects in the large-scale data sets, such as ships
and windmills, are very small scaled, weak featured, and
easily affected by the context and low-quality data.
There are data sets that are very challenging and suitable
to develop the detectors for small, weak object detection.
VEDAI [17], 3K vehicle [16], and COWC [39] are three
relatively small-scale data sets used for vehicle detection. As
shown in Figure 7, their images have VHR (up to about
12.5 cm) and their objects, which are fixed to a range, are
beneficial for developing and testing a model to detect
DECEMBER 2021 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
small objects. They can supplement the large-scale data
sets mentioned previously. However, some critical limitations
in these data sets are that the object categories are few,
including only cars or airplanes, and the image quality is
high, which is not consistent with actual scenarios. VHR
data sets with more categories, variations, and challenges
need to be developed further.
EVALUATION METRICS
There are two categories of metrics for evaluating detector
performance: detection speed in frames per second
(FPS) and detection accuracy in precision, recall,
and average precision (AP). FPS is a metric used to express
how fast the detector is; it means the number of
image FPS that the detector can process. For example,
if the time needed for a detector to analyze a standardscale
image is 0.04 s, its detection speed is a frame rate
of 25 FPS.
For a given input image I, the outputs of a detector are
the predicted results {( ,)}bc p,jj j
M
j =1 (indexed by the object
order j; M is the number of predicted detections) of the BB
b ,j
predicted label c ,j and confidence score .pj
truth boxes are {( ,)}BC N
C .k {( ,, )}bc pjj j
the number of ground-truth boxes) of the BB Bk
M
j =1 are greedily matched to {( ,)}.BC
N
kk k=1
For given a confidence threshold t and a intersection over
union (IoU) threshold
e , a predicted result (, ,)bc pjj j
ground-truth box (, ),BC and pj
kk
is computed as
Io (, )Ub B
jk =
is set
as a true positive (TP) if the following criteria are met:
◗ The predicted label cj
◗ The IoU value between the predicted BB bj
ground-truth BB B ,k
Io (, )Ub Bjk
area bB
area bB
_
_
jk
jk
(
' ,
i
i
is equal to the label Ck of a
is greater than t.
and the
Io (, ),Ub Bjk is larger than ,e where
(1)
and the symbols of + and , denote the intersection and
union, respectively. The value of e is generally set to 0.5.
Otherwise, the predicted result is regarded as a false-positive
(FP) sample.
Precision is the proportion of correct detection instances
out of the total detection results predicted by the detector.
Based on the calculations of the TP and FP results, it can be
computed by
Pt TP FP
TP
() =
+
.
es indicated by a detector. It can be formulated by
()
Rt N
= TP
,
(2)
Recall is defined as the proportion of all positive instanc(3)
where
N is the number of ground-truth boxes.
Precision and recall can drive AP, which is the metric
most used in recent works. AP is usually computed for each
class separately. The precision, P(t), and the recall, R(t), can
17
The groundkk
k=1 (indexed by the order k; N is
and label
IEEE Geoscience and Remote Sensing Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2021
