IEEE Computational Intelligence Magazine - May 2021 - 34

B. Deep and Hierarchical Models

Recently, deep learning methods, especially CNNs, have
achieved promising results in EEG signal decoding research.
For instance, Schirrmeister et al. [3] introduced Shallow
ConvNet, Deep ConvNet, Hybrid ConvNet, and Residual
ConvNet. They evaluated how well various proposed CNNs
decoded MI. Ko et al. [18] also proposed a novel CNN architecture which is inspired by a recurrent convolutional neural
network [38] for MI classification, deep recurrent spatio-temporal neural network (RSTNN). Despite great success in deep
learning-based MI decoding methods, some limitations in
subject-dependent MI-based BCIs still exist [27]. However,
owing to its great benefits such as zero-calibration, some pioneering research [17], [27] proposed deep learning-based subject-independent MI-BCIs. Zhang et al. [17] proposed a
combined recurrent neural network (RNN) and CNN for
extracting spaito-temporal feature representation. Kwon et al.
[27] exploited multi spectral-spatial features (SSFR) using
spectral-spatial filtering and CNN and experimented on the
large-scale database.
While a standard CCA [9] has obtained state-of-the-art
performance in SSVEP-based BCI, Kwak et al. [23] developed
a CNN for SSVEP feature representation learning. These
authors simply combined spatial and temporal convolution to
enable the system to learn data patterns in the latent space,
thereby correctly generalizing EEG signal features. Meanwhile,
Waytowich et al. [22] applied EEGNet [4] to the SSVEP paradigm and achieved a higher performance than that of the standard CCA.
Supratak et al. [20] developed a deep neural network for
sleep stage detection. More precisely, they combined a CNN
for representation learning and a recurrent neural network for
sequential residual learning. Furthermore, they trained the
deep learning model in two separate steps, optimizing the
model by individual pre-training and fine-tuning. In the
meantime, Gao et al. [26] proposed an EEG-based spatio-temporal convolutional neural network (ESTCNN) for driver
fatigue evaluation. The ESTCNN simply convolved the bandpass filtered EEG to represent temporal dependencies and flattened the extracted features for spatial features fusion. Lastly,
densely connected layers were used for the identification of a
user's condition.
To detect a seizure type, Asif et al. [24] proposed a multispectral deep feature learning using a deep CNN, SeizureNet.
They transformed the EEG signals to spectral space using
saliency-encoded spectrogram generation and fed the extracted
spectral features to a deep neural network. In the meantime,
Emami et al. [25] independently proposed another CNNbased approach for detecting seizure onset. They band-pass filtered and segmented the input EEG patterns, and then used a
deep CNN for classification.
Recently, Lawhern et al. [4], [22] proposed a novel CNN,
the so-called EEGNet. Unlike other linear or deep learningbased methods, the EEGNet classified various EEG paradigms
using a single architecture, i.e., not specifically tuned for different

34

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021

paradigms. Furthermore, Lawhern et al. used a separable
convolution [19] for parameters reduction method.
On one hand, the deep and hierarchical models decoded
the EEG signals well without any custom feature extraction
stage for their respective paradigms [3], [18], [20], [23]-[26] or
even various paradigms [4], [22]. On the other hand, the deep
CNNs extracted the EEG features at a sequential level using
stacked convolutional layers without exploiting multi-scale
spectral representation. Conversely, the proposed method
exploits multi-scale spatio-spectral-temporal features irrespective of the input EEG paradigms.
III. Methods

In this section, we propose a deep multi-scale neural network
(MSNN), which can represent EEG features from different
paradigms by exploiting spatio-spectral-temporal information at
multi-scale.
A. Multi-Scale Neural Network

As mentioned previously, a FBCSP [11] is one of the most successful models to exploit multi-scale EEG signal features, especially for MI. Thus, many successful MI EEG signal decoding
algorithms [3], [13] or even other paradigm classification algorithms [4] were inspired by the FBCSP model. In this study,
the proposed MSNN also learns multi-scale feature representations. However, the network automatically learns from data
through discriminative multiple spectral filters, rather than
manually defining multi-frequency bounds as in FBCSP. Basically, our proposed method consists of three types of blocks:
(1) a spectral-temporal feature representation block, (2) a spatial
feature representation block, and, (3) a classification block, as
depicted in Fig. 2.
First, in the spectral-temporal feature representation block,
stacked convolutional layers extract EEG data spectral-temporal
features, similar to the existing deep learning-based EEG classification methods [3], [4]. However, the proposed model
exploits the intermediate activations for gathering multi-scale
spectral information. Then, the spatial feature representation
block discovers spatial patterns from the extracted multi-scale
features. Finally, these multi-scale spatio-spectral-temporal features are concatenated, pooled, and fed into the densely connected layer for classification.
B. Spectral-Temporal Feature Representation Block

Given an input EEG data x, we reshape it in the form of
[n c, n T , 1], i.e., x ! R nc # nT # 1, where n c and n T denote the
number of electrode channels and timepoints, respectively.
In the MSNN, the input EEG data is temporally convolved
in a channel-wise manner by a spectral convolutional layer to
expand the number of feature maps. Thus, the activated features
have the form [n c, nlT , F0], where nlT = n T - ( fs /2) + 1 ( fs and
F0 are the sampling frequency and the feature map dimension
for the first temporal convolution layer.). The main benefits of
using a separable convolution [4], [19] are a significant reduction of tunable weights, in the model and more importantly, an
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
