IEEE Geoscience and Remote Sensing Magazine - March 2022 - 304

where TN represents the true negative classifications, TP
is true positive classifications, FP represents false positive
classifications, and FN denotes false negative classifications.
In a balanced data set, the accuracy score can give an
overall metric for model performance, but supplementary
measurements are needed to understand how well a model
predicts each class in a data set [89]. The three additional
metrics of precision, recall, and the F-1 score were calculated
to assess the results of each model by using scikit-learn.
Precision is a measure of the positive predictive value of
a model or how many of the samples that were predicted
as locusts were correct. Recall is a measure of the fraction
of actual locusts present in classifications identified by a
model. A recall score of one means a model did not make
any false negative predictions. The F-1 score is the harmonic
mean or weighted average of the precision and recall metrics
and is the best measure to use when comparing results
of multiple models and data sets [91]. The F-1 measure applies
a penalty to a classifier with imbalanced precision and
recall scores. Low values are given more weight when calculating
the mean to highlight disparities between precision
and recall scores [86].
The MaxEnt model predicts suitable habitat by using
only presence data, so it requires different metrics to evaluate
its classification performance than the multiclass metrics
of precision, recall, and the F-1 score. The area under
the receiver operating curve (AUC) metric is harnessed to
measure how well a model can predict compared to random
chance [97]. When performing a binary classification
in MaxEnt and with scikit-learn, the AUC score is based
solely on the performance of the positive class. Therefore,
the AUC measures whether a model classifies locust presence
more accurately than a random prediction rather than
whether the model can accurately distinguish presence
from absence locations. An AUC value of 0.5 indicates that
a model predicts no better than a random one, whereas
AUC values of >0.75 are considered a " good fit " [11], [86].
RESULTS
Several machine learning algorithms were used to evaluate
the seasonal and annual data sets for each study area.
For all models, 70% of the samples from each data set were
used for training, and the remaining 30% were withheld
for testing predictive performance. Random seeds were
used where appropriate to ensure the reproducibility of the
results. In the MaxEnt application, the sample subset mode
and automatic limiting of feature types (linear, quadratic,
and so on) based on the complexity of the samples were
selected. For the models run using scikit-learn, the training
data sets were evaluated using fivefold cross validation in
GridSearch to determine the best parameters and estimators.
The optimal model was then tested using the proximal
data set that was withheld during model training, and the
corresponding validation metrics were calculated.
Overall, the model with the highest accuracy score
in both study sites was the DNN (Tables 4 and 5). It performed
well across multiple seasons and while using all
three absence types. The kNN preformed comparably in
both regions and showed similar robustness across data
sets. Neither tree-based model was able to achieve comparable
accuracy scores; the DT was the least successful. It
was less able to establish a clear boundary between the two
classes across multiple seasons and absence types, which
may have been the result of the model overfitting to the
training data despite the PCA dimensionality reduction. In
general, the accuracy percentages were higher when using
TABLE 4. TEST ACCURACY CLASSIFICATION RESULTS FOR THE NIGER STUDY AREA.
ENVIRONMENTAL NORMALIZATION
GEOGRAPHIC EXCLUSION
CLASSIFIER ANNUAL WINTER
kNN
DT
0.91
0.9
SVM
RF
DNN
0.87
0.92
0.93
0.95
0.95
0.93
0.96
0.94
SUMMER
0.93
0.91
0.94
0.92
0.95
ANNUAL WINTER
0.84
0.78
0.86
0.78
0.84
0.95
0.93
0.88
0.94
0.91
SUMMER
0.94
0.91
0.92
0.92
0.94
RANDOM
ANNUAL WINTER
0.78
0.72
0.77
0.76
0.78
0.87
0.83
0.75
0.86
0.86
SUMMER
0.77
0.71
0.81
0.78
0.81
TABLE 5. TEST ACCURACY CLASSIFICATION RESULTS FOR THE SUDAN STUDY AREA.
ENVIRONMENTAL NORMALIZATION
GEOGRAPHIC EXCLUSION
CLASSIFIER ANNUAL WINTER
kNN
DT
SVM
RF
DNN
0.92
0.86
0.87
0.91
0.91
0.97
0.94
0.95
0.96
0.95
SUMMER
0.99
0.96
0.99
0.96
0.99
ANNUAL WINTER
0.82
0.78
0.83
0.79
0.84
0.99
0.97
0.95
0.98
0.99
SUMMER
0.79
0.78
0.74
0.78
0.81
RANDOM
ANNUAL WINTER
0.8
0.78
0.82
0.85
0.8
0.88
0.85
0.79
0.88
0.89
SUMMER
0.77
0.72
0.7
0.72
0.72
304
IEEE GEOSCIENCE AND REMOTE SENSING MAGAZINE MARCH 2022
IEEE Geoscience and Remote Sensing Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Geoscience and Remote Sensing Magazine - March 2022
