IEEE Computational Intelligence Magazine - May 2021 - 39

spatial, with a squaring nonlinear activation, an average pooling,
and a logarithmic activation. The Deep ConvNet [3] has five
convolutions, temporal and spatial, and three additional temporal convolutions. The RSTNN [18] is also used for these
experiments. This network consists of three recurrent convolutional layers, and each recurrent convolutional layer has three
recurrent temporal convolutions [38] with a spatial convolution. Additionally, we also conducted experiments on two
combined deep neural networks, 'parallel' and 'cascade' convolutional recurrent networks (Parallel CRN and Cascade CRN)
proposed by Zhang et al. [17]. In both models, EEG signal vectors were converted to 2D meshed data according to the
acquired electrode map. In the parallel structure, a three-layer
CNN and a two-layer RNN simultaneously extracted spatial
and temporal features respectively, and a concatenation is used
for the feature decomposition. Furthermore, in the cascade
structure, a three-layer CNN extracts spatial features of mesh
data at each timepoint, and the extracted features are fed into a
two-layer RNN to represent temporal information. Note that
we only conducted experiments on the MI and SSVEP datasets [8], [40] due to the lack of electrode channels of other paradigms to generate 2D meshed data.
6) Deep Neural Networks-Steady-State Visually Evoked
Potentials
For the SSVEP decoding experiment, we exploited another
version of EEGNet for SSVEP EEG [22]. We used different
kernel sizes for this EEGNet as Waytowich et al. proposed. The
SSVEP classification performance estimated by this version is
marked by @ in the classification table.
7) Deep Neural Networks-Drowsiness
The ESTCNN [26] which is proposed for mental fatigue classification has three core blocks. Each block in the ESTCNN
consists of three temporal convolutions with a max pooling
layer, with the exception of the last block that uses an average
pooling layer instead of the max pooling.
8) Deep Neural Networks-Multi-paradigm
Finally, we also implemented the EEGNet [4] in our study. As
previously mentioned, we used different kernel sizes for two
different EEGNets, [4] and [22]. Nevertheless, the basic architecture of the network was the same for various EEG paradigms, having a temporal convolution, depthwise spatial
convolution [19], and separable temporal convolution [19].
9) Proposed Multi-Scale Neural Network
While training our proposed network, depicted in Fig. 2, we
set a mini-batch size of 16, an exponentially decreasing learning rate (initial value: 0.03, decreasing ratio per epoch: 0.001),
and an Adam optimizer. For the first temporal convolution, we
used a conventional temporal convolution with the kernel size
of (1 # fs /2) and F0 = 4. Furthermore, we used three spectraltemporal feature representation convolutions, i.e., N = 3, and
set T1 = 100, T2 = 60, and T3 = 20 with F1 = 16, F2 = 32,

and F3 = 64. Then, for the spatial feature representation block,
we used three spatial convolutions because the number of spatial convolutional layers must be the same as the number of
spectral-temporal separable convolutional layers. The proposed
method used different kernel sizes for the SSVEP dataset, similar to the EEGNet [22] due to the fact that SSVEP EEG data is
created by target frequencies [8], [9]. For the KU-SSVEP dataset [8], we set T1 = 20, T2 = 10, and T3 = 5 for the spectraltemporal feature representation block, and used the same
settings for the others. The SSVEP classification performance
estimated by this method is marked by A. Additionally, batch
normalization was performed after every convolution. Finally,
for the classification block, all activated features from the spatiospectral-temporal block were concatenated and fed into the GAP
[39] layer. Then, after flattening, the multi-scale features were linearly mapped by a dense layer. In this proposed network, a
leaky rectified linear unit (ReLU) activation function, a L1-L2
regularizer (, 1 = 0.01 and , 2 = 0.001), and a Xavier initializer
[41] are used for all tunable parameters except for the final
decision layer that is activated by a softmax activation function
instead of a leaky ReLU. We selected model components that
demonstrated the best performance for validation, i.e., model
selection samples, as mentioned previously.
C. Experimental Results

In this section, we summarized all experimental results in Table II.
Furthermore, we performed two-tailed Wilcoxon's signedrank test to estimate p-values of competing baseline models
and our proposed MSNN, comparing performance differences statistically.
1) Motor Imagery (subject-dependent)
Our proposed network clearly outperformed other baselines for
MI EEG signal decoding. Importantly, the proposed network
achieved a higher accuracy than the methods designed specifically for MI classification: CSP [1], FBCSP [11], Shallow
ConvNet [3], Deep ConvNet [3], and RSTNN [18]. With this
clear improvement in accuracy, we could expect that our proposed
method is one step closer to MI-based BCI commercialization.
2) Motor Imagery (subject-independent)
By experimental results summarized in Table III, we conclude
that our proposed MSNN is well-generalized not only in the
subject-dependent manner, but also in the subject-independent manner. Based on this promising result, we also conclude that our method can be used for the zero-calibration
in MI-based BCIs.
3) Steady-State Visually Evoked Potentials
Our proposed MSNN achieved a slightly lower performance
than CCA [9], Deep ConvNet [3], and EEGNet [22] in the
SSVEP classification. However, the difference in performance
between our MSNN and the other three baselines, CCA [9],
Deep ConvNet [3], and EEGNet [22], was reasonably small and
the proposed method performed with a credible accuracy score.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

39
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
