IEEE Geoscience and Remote Sensing Magazine - December 2022 - 199

Evaluation results are shown in Table 6 with trackers' backbones
and model tags. Four kinds of evaluation methods
(average IoU, average EIoU, average recall score, and average
ENUS1) are calculated to make complete comparisons of
trackers in different aspects, and these methods have been
introduced in the " Tiny Objects Enhanced Measurements "
section. Besides, additional statistics for different classes
and difficulties are also calculated to help understand the
overall performance. Although DLVs and SVs have different
image feature space, some experimental trackers still make
a relatively high tracking performance.
The top three trackers are labeled with colored fonts for
each evaluation method, while the top 4-10 trackers are labeled
with bold black fonts. From the tracking evaluation
results, in general, the best performance tracking model is
the SiamRPN++ with the backbone of ResNet-50, which
has achieved most of the top three ranks for all evaluation
methods. SiamAttn and SiamMask have achieved top three
recall scores from 0.6366 to 0.6778, containing the targets'
pixels as much as possible.
Other trackers also achieved good results for different
evaluation methods, which are summarized as follows:
◗ IoU: Using the original IoU score, SiamRPN++, DaSiamRPN,
and SiamDW achieved the best score, while other
trackers, such as TrDiMP; TrTr; SiamFC++ (AlexNet-Online,
GoogLeNet-TrackingNet-full); DCFST; SiamMask
(default); and ECO, made a relatively good prediction.
◗ EIoU: With EIoU, all of the scores have a small decrease
for adding on the distance factor. The D3S, SiamMask,
SiamMask-E, RT-MDNet, ECO, SiamAttn, DCFST, SiamDW,
and ATOM trackers have a decrease of about 3%
or more than 3%, which means these trackers have a
worse position prediction than others.
◗ ENUS: By using the ENUS1 evaluation method, the rank
has some differences from the rank with the IoU score.
TrDiMP (ResNet-50) reached the third with the score of
0.4238, while SiamGAT (GOT10k) and SiamDW (CIResNet22-RPN)
reached to the first (0.3585) and second
(0.3375) in the robustness test, respectively.
Evaluations for different classes and difficulties use
ENUS1 to obtain high-precision and comprehensive assessments.
SiamRPN++ (ResNet-50-Default) made the best
performance on all of the classes (ships is the first, and the
others are second), and all of difficulty levels (first). PrDiMP
(ResNet-50, ResNet-18), TrTr (ResNet-50-Online), and SiamDW
(CIRestNet-FC) have the best performance on tracking
airplanes, large vehicles, and vehicles, respectively. The
average and best scores are listed at the end of each column.
Obviously, tracking scores are positively correlated with OSs
but negatively correlated with scene difficulty levels. (i.e., the
small target size with lots of frames occurs with occlusion or
blur, making worse tracking results and lower ENUS1.)
To make further detailed comparisons on the proposed
result table, we draw the OPE success plot with ENUS1 measurements
for the top 25 tracking models for each figure,
referring to Figures 14 and 15, including an overall plot, 10
DECEMBER 2022 IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE
tracking challenges, four classes, six sequences, and three
difficulty levels. From the graphs with 10 tracking challenges,
the trackers show their different capacities.
DS and BCL usually occur in vehicle-like objects in SVs.
The traditional methods MCCT-H and ECO-H make the best
tracking score, so the artificial features show their advantages
of representing tiny objects. The SiamRPN++ also obtains
good results because of its feature pyramid structure. Background
change (BCH) is examined in environmental change.
SiamRPN++, MCCT-H, and ECO-H make the best track in
this situation, so they have the best feature representations.
For different occlusion situations, SuperDiMP and KeepTrack
perform better in LTO, while TrDiMP, SiamRPN++, and
ECO-H obtain similarly best results in STO. However, when
CO occurs, SiamRPN++ is the steadiest tracker.
IPR is another main challenge because objects may have
significant appearance deformations when they change their
orientation. At the same time, illumination variation also
causes significant appearance changes because the object's
color can be changed completely. SiamRPN++, MCCT-H,
and SiamDW are the best
trackers in these two situations,
and ECO-H also has a
good performance for illumination
variation.
The top five trackers for
ND are MCCT-H, KCF, MDNet,
TrDiMP, and TrTr. These
trackers have strong antinoise
capacities to keep track of the
object influenced by smog,
sandy weather, or moving
clouds. The top five trackers for SM are SiamDW, MDNet,
MCCT-H, SiamRPN++, and ECO-H. SM usually happens
when the object is waiting for the traffic light or turning
at an intersection. Therefore, this challenge is more complicated
than the others, and these five trackers have more
robust adaptive capacities than others.
To better understand these trackers' performance on all
tracking challenges, a radar graph for each tracker is shown
in Figure 16. The larger area indicates that the tracker has
more comprehensive capabilities in tracking objects in SVs.
These radar graphs are further details of the challenge plots
in Figure 14, which can help researchers study the characteristics
of each model.
INFERENCE SPEED EVALUATION
To test the inference speed on different GPU devices, we select
30 sequences with a simple tag from six videos as the minidataset,
and these sequences include all kinds of objects. Test
results are provided in Table 7. Four GPU devices are tested
on a mini-dataset, including three types of GPUs: TITAN X,
GTX 1080, and GTX 2080; also, the NVIDIA TX1 is used to
simulate the onboard equipment. From the test results, only
the SiamRPN++ with AlexNet (the fps is 12.291 > 10) reaches
the real-time inference on mobile devices. Due to platform
199
THIS COMPREHENSIVE
EVALUATION RESULT CAN
BE TREATED AS A GOOD
REFERENCE FOR DESIGNING
MORE EFFICIENT AND
ROBUST TRACKERS FOR SVs.
IEEE Geoscience and Remote Sensing Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - December 2022
