IEEE - Aerospace and Electronic Systems - July 2021 - 55

Vakil et al.
Table 7.
RF Accuracy Comparison
Situation
Comparison of simulations 9 and
10, differentiating between
detecting one vehicle and two
vehicles
Comparison of simulations 2, 9, and
10, differentiating between
detecting one vehicle, two vehicles,
and three vehicles
Combined data from simulations
2,4,9,10,12, and 13, differentiating
between detecting one, two
vehicles, and three vehicles
different histograms. The addition ofall these training and testing
data reduced the accuracy oftheNN to 83%.
As shown in Table 8, the overall accuracy of different
Accuracy
95%
92%
sensors on their own is unsatisfactory, with EO only managing
to score a 72% accuracy against the ground truth
and RF only reaching 83% accuracy when trained and
tested with large number of different scenarios. With
FERNN, the accuracy can reach a much higher level of
95%, when the EO frames, the data of P-RF sensors
located at nadir, north, and west are all fed into the NN.
Based on the results of the feature-level fusion, a sig83%
ensemble
to make the final prediction with the available
data. Besides soft and hard voting, the data from the feature-level
fusion experiments, the standalone EO and
standalone RF NNs, were used to implement decisionlevel
fusion. The predictions for each of the individual
models were fed into an SVM model that used the
concatenated values. Besides SVM, a NN model used the
same prediction values to classify the dataset.
RESULTS
The trained standalone EO NN alone could reach 91%
accuracy in determining the current number of detected
vehicles based on the available image provided. However,
when tested against the ground truth, with the knowledge of
a vehicle moving outside of the camera's angle or obscured
by local foliage, the overall accuracy of the NN would
decrease to only 72%. Similarly, the standalone P-RF NN,
couldonlyreachanaccuracyof 83% whentakinginto
account all the histograms for the selected simulations,
where scenario 1 describes a ground truth of one vehicle,
scenario 2 describes a ground truth of two vehicles, and scenario
3 describes a ground truth of three vehicles.
As seen in Table 7, the accuracy of the standalone P-RF
NNwas significantly higher when trained with a small number
of simulations. When comparing simulations 9 and 10 to differentiate
between one vehicle and two vehicles, the NN could
perform at 95% accuracy. However, when the number ofcategories
increases to 3 and the NNwas trained with three different
simulations, the performance decreased to 92% accuracy.
In this situation, histograms formed by the P-RF feature extraction
are still unique enough to ensure an acceptable accuracy.
In order to ensure that the training data were robust enough for
the purposes of vehicle detection, six simulations were combined
in the training process, each ofwhich containing visually
JULY 2021
nificant increase in accuracy was dependent on the number
of feature sources for training available. As seen in
Table 9, the F1 score for using only the EO and Nadir
SIGINT is 80%. Similarly, the F1 scores for the EO and
North and EO and West are at 78% and 82%, respectively.
Based on the results presented in Tables 10-12, the accuracy
of the NN only reaches satisfactory values when all
four sources of SIGINT (see Table 13) and the corresponding
frames are fed into the training
F1Score ¼
2 Precision Recall
Precision þ Recall
Precision ¼
Recall ¼
True Positive
True Positive þ False Positive
True Positive
True Positive þ False Negative
:
(1Þ
(2Þ
(3Þ
Table 8.
Accuracy Comparison Between Standalone Modalities
and Feature Level Fusion
Situation
Standalone EO
Standalone RF
83%
Feature Level Fusion Architecture 95%
Table 9.
EO and Nadir sigint Fusion
Scenario
1
2
3
Accuracy
Macro AVG 0.80
Weighted AVG 0.80
IEEE A&E SYSTEMS MAGAZINE
Accuracy
72%
Precision Recall F1-Score
0.70
0.87
0.83
0.72 0.71
0.79 0.83
0.88 0.85
0.80
0.80 0.80
0.80 0.80
55
IEEE - Aerospace and Electronic Systems - July 2021

Table of Contents for the Digital Edition of IEEE - Aerospace and Electronic Systems - July 2021
