IEEE Systems, Man and Cybernetics Magazine - January 2022 - 12

easily separable for three levels of cognitive loads. Similar
observations hold for the remaining participants and have
been omitted for brevity.
Figure 3 shows the NASA-TLX scores of the participants
for all load levels in the experiment. It can be seen that the
MATB-II configurations used to manipulate the three load
levels has been successfully picked by the subjective score
making workload discriminations, which cements the
experimental process as well.
Result and Discussion
Figure 4 shows the accuracy of TL and four baseline methods
for seven participants. This work considers LDA,
10
20
30
40
50
60
70
80
Low Medium High
Cognitive Load Level
Figure 3. The NASA-TLX scores of the participants.
100
100
logistic regression, NNs, and RF as the out-of-the box
classifiers. Each of these classifiers are used in baseline
method (i.e., model trained and calibrated using data set
from individual users) as well as the proposed TL method
(i.e., model trained using crowd sourced data sets). RF is
included in both baseline and proposed methods to investigate,
and attribute, classification performance.
The TL method uses an RF with training data from
other users, except for one in which the performance will
be tested. The baseline methods are different machine
learning models with both training and testing data from
the user on which the performance is tested. For example,
the baseline LDA for User 1 represents an LDA model
trained with data from User 1 and tested the performance
on User 1. In contrast, the TL model is trained on data
from User 2 to User 7 but tested on User 1. In this study, we
used the same architecture and learning parameters for
RF in TL and baseline model. This implies the testing of
the TL method involves more samples in the test set than
the baseline methods.
The accuracy versus user ID graph (Figures 4-7) represents
the result of 10-fold cross-validation with ten repetitions
(e.g., a total of 100 experiments on learning and
testing). It is prominent from this figure that the TL method's
accuracy is the highest compared to all of the baseline
models. Also, the performance variations for
individual participants are shallow, and all are slightly
above or below 98%. Among all of the baseline methods,
RF classifiers give the best performance for all seven participants.
For users 1, 6, and 7, it is almost 96%, and for
70
60
70
60
12 34 56 7
User ID
Baseline_LDA
Baseline_RF
Baseline_LDA
Baseline_RF
Baseline_NN
Baseline_Logistic
Baseline_NN
Baseline_Logistic
TL
1234567
User ID
Baseline_LDA
Baseline_RF
Baseline_LDA
Baseline_RF
Baseline_NN
Baseline_Logistic
Baseline_NN
Baseline_Logistic
TL
Figure 4. The distribution of accuracy over 100 runs
(10-fold CV, repeated 10 times) for all participants
in the experiment. Higher is better. The proposed
method, as described in the " TL Method " section,
is visible at the very top with tiny variance in the
corresponding boxplot.
12 IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE January 2022
Figure 5. The precision of the TL and baseline
methods. The figure is drawn using methods
described in the " TL Method " section. The precision
for all models varies based on the user, which reflects
the subjective nature of EEG data. However, the TL
method remains much better than the baseline
methods. Micro averaging is used for precision, recall,
and f1-score computation.
Accuracy (%)
NASA-TLX Score
Precision (%)
IEEE Systems, Man and Cybernetics Magazine - January 2022

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - January 2022
