IEEE Computational Intelligence Magazine - May 2021 - 41

proposed method using different methodologies, activation pattern maps [30] and t-SNE plots. Additionally, we observed a
practical use for the proposed method, especially for drowsiness
and seizure detection experiments.
A. Multi-Scale EEG Feature Extraction

To demonstrate the multi-scale information capture ability of
our proposed method, we estimated and plotted PSD values
and relevance scores for MI EEG samples. Specifically, we estimated PSD values for subject 48 and 52 in the GIST-MI dataset [40]'s EEG samples from channels on the motor cortex.
Additionally, we calculated relevance scores for those subjects
using a layer-wise relevance propagation [42]. In our results, all classification methods evenly demonstrated good generalization
(baselines: ~85% and proposed: ~90%) for subject 48, whereas
only the proposed method achieved superior performance for
subject 52 (baselines: <65% and proposed: ~80%). As Fig. 4
shows, subject 48's EEG samples are highly activated at the μ
range, while subject 52's samples do not show any clear trend at
the μ range, but in a wider range. In the meantime, in our classrelevance score visualization (Fig. 4), we could observe the
relation between the PSD characteristics of the input EEG and
the features' relevance to the output class decision. That is, for
subject 48 (Fig. 4(a)), features of G ^f 1SSTh showed higher classrelevance scores than those of other scales G ^f 2SSTh and
G ^f 3SSTh. Meanwhile, for subject 52 (Fig. 4(b)), features of all
scales, i.e., G ^f 1SSTh, G ^f 2SSTh, and G ^f 3SSTh showed similar rele-

vance scores, meaning that most of those features were involved
in making the class decision.
From this phenomenon, we can conclude that our proposed
MSNN can capture important features on the multi-scale range,
not only in the frequency of interest. In other words, while
other existing methods gather spatio-spectral-temporal information at the sequential level, the proposed network exploits
multi-scale features, thereby improving learning ability.12
B. Activation Patterns

Earlier, Haufe et al. [30] proposed an activation pattern which is
based on a forward-backward modeling in signal processing. The
activation pattern method [30] provides a way to interpret
weight matrices in multivariate neuroimaging, as presented in
the signal processing literature.
The proposed method, clearly, decodes the input EEG signal to the corresponding label, i.e., inferring a user's intention
or condition from an observed EEG pattern. Therefore, it is a
backward process computational model. Hence, for a concrete
and meaningful understanding of learned layers, it is essential to
reverse this backward process model to a forward process.
Finally, in this work, we estimated and visualized the activation
patterns of the learned weights shown in Fig. 5(a). We extracted the spatial convolutions of Shallow ConvNet [3], Deep
ConvNet [3], RSTNN [18], EEGNet [4], and the proposed
12

Randomly selected additional results are reported in Supplementary C.

µ V 2/Hz (dB)

158

g(f1SST)

g (f2SST)
SST )
g (fconcat
of Subject 48

Subject 48
135

5

10

15 20 25 30
Frequency (Hz)

g (f3SST)

35

40

µ V 2/Hz (dB)

158

g(f1SST)
135

10

15 20 25 30
Frequency (Hz)

g (f3SST)
SST )
g (fconcat
of Subject 52

Subject 52
5

g (f2SST)

35

40

FIGURE 4 PSD curves (left) and relevance scores [42] (right) for subject 48 (top) and subject 52 (bottom) from the GIST-MI dataset [40]. For the
PSD curves, the solid red line and the shaded region exhibit the mean and standard deviation of PSD values of all trials, respectively. For the relevance scores, G (f kSST) denotes the global average pooled features of the kth spectro-temporal layer and the deeper red color denotes a higher
relevance score. We observed that our proposed MSNN concentrates features from the lower frequency range for subject 48 and a wider range
for subject 52 from the relevance scores. This observation supports that our proposed MSNN is able to represent neurophysiological patterns
from the input EEG signals.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

41
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
