IEEE Computational Intelligence Magazine - May 2021 - 43

model. Then, we estimated activation patterns
From this phenomenon, we can conclude that our
and visualized them in a topological manner.
We do not estimate ESTCNN [26] activation
proposed MSNN can capture important features on the
patterns because the ESTCNN [26] does not
multi-scale range, not only in the frequency of interest.
have any spatial feature representation layers
and the visualized patterns are estimated by
the first subject's first fold data in the GIST-MI dataset [40].
first session data in the KU-SSVEP dataset [8], and used a
Finally, we normalized the activation patterns in [0, 1] range
learning rate of 200, a perplexity of 10 for the t-SNE calculabefore visualization.
tion, and visualization.
In this investigation, we observed right-lateralized brain
From these visualized represented features, we could
SST
activation/deactivation patterns, and the same patterns in the
observe that G (f concat
) is more class-discriminative than the
left hemisphere when a user imagined the movement of the
other intermediate features. Additionally, we observed a
left hand and right hand respectively. Since there is no golden
trend, which demonstrated that a feature learned by a deeprule to measure a discriminability, we exploited a cosine simier layer is more disentangled than others learned by shallarity score.13 Based on our estimation, our proposed method
lower layers.
achieved the smallest cosine similarity (0.0047) among the
other baselines (0.0224 + 0.2750), thus we can conclude that
D. Mental Fatigue Classification
our proposed MSNN has relatively clearer patterns than the
For the application analysis of drowsiness detection, we visualother models; in other words, our method thoroughly repreized confusion matrices that were estimated by the experimensents input EEG signal spatial features.
tal results of the SEED-VIG dataset [36] in Fig. 5(c). Since the
labels that identify the mental status were decided using the
PERCLOS levels [36], the label at the boundary of the two
C. t-SNE Visualization of Learned Features
classes may not be accurate. In this respect, we can conclude
To validate the representation ability of the proposed network,
that the proposed method is useful for drowsiness state detecwe plotted t-SNE transformed learned features shown in Fig.
tion because false detections predicted by the proposed method
5(b). Specifically, we exhibited extracted features from test
are mostly at the boundaries between classes, e.g., the 'awake' vs.
SSVEP EEG samples from the first, second, and third spatio'tired' or 'tired' vs. 'drowsy' case. In addition, for practical applispectral-temporal feature representation layers, i.e., f 1SST, f 2SST,
cation, it is essential to detect the drowsy state accurately to
and f 3SST (first three figures in Fig. 5(b)). Then, we also
SST
avoid unexpected situations, such as a car accident. The prodepicted the final learned feature, i.e., G (f concat
). These interposed method achieved the highest and most promising result
mediate features f 1SST, f 2SST, and f 3SST are temporally pooled
SST
for detecting drowsiness among other baselines, i.e., it achieved
just for visualization like G (f concat
). We used the first subject's
the highest precision score for identifying the drowsy state.
Therefore, we can also expect that our proposed method can be
13
See 'Supplementary F: Additional Results for Activation Pattern' for additional
applied in real-world situations.
results of the activation patterns and analyses in the quantitative standpoint.

Probability of
Whether Input EEG Is Ictal or Not

1
Seizure On/Offset
Shoeb and Guttag
Shallow ConvNet
Deep ConvNet
RSTNN
ESTCNN
EEGNet
MSNN

0.5

46
Relative Time (s)

(d)
FIGURE 5 (Continued) Investigation of learned weights (Fig. 5(a)) and represented features (Fig. 5(b)), and inspection of the practical usage of
the proposed network (Fig. 5(c) and 5(d)). (d) Changes of probabilities estimated by comparable baselines, and the proposed method. These
plots demonstrate the probability of whether input EEG is ictal or not. Two dot-dashed lines (magenta) denote the seizure onset and ending,
respectively, labeled by Shoeb [12].

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021