IEEE Computational Intelligence Magazine - May 2021 - 33

Finally, interpreting a learned stacked CNN from a neurophysiologically appropriate standpoint [30] is quite complicated
because the CNN identifies complex patterns of data in latent
space, making a direct explanation difficult.
In this study, we propose a novel deep learning-based BCI
method to mitigate the previously discussed difficulties. We
conduct exhaustive experiments on various EEG paradigms
and also perform diverse analyses. According to our observations, the proposed deep neural network achieved comparable performances in various paradigms with reasonable
neurophysiological interpretations. In this regard, we believe
that the proposed network can be used as a starting model
and/or a backbone network for any new BCI protocols,
experiments, and paradigms. The main contributions of our
study are as follows:
❏❏ First, we propose a novel CNN architecture that is applicable independently from the input paradigm or types of
EEG, and can represent multi-scale spatio-spectral-temporal
features.
❏❏ Second, the proposed method achieved reasonable performance on five different datasets for four different paradigms
(two for active BCIs and two for passive BCIs). The proposed method outperformed or was similar to state-of-theart linear and deep learning methods, which were
individually designed for each specific paradigm.
❏❏ Third, the proposed method showed robust performance on
three different paradigms (MI, drowsy, and seizure) without
specific tuning of network architecture, e.g., kernel sizes.
❏❏ Lastly, we analyze the proposed network using a variety of
techniques.
The rest of this paper is organized as follows: Section II
reviews previous research on various EEG representation learning via linear model-based or deep learning-based methods. In
Section III, we propose a novel and compact deep CNN that
classifies multi-paradigm EEG by representing multi-scale spatio-spectral-temporal features. Section IV presents experimental
settings and results by comparing the proposed method and
comparable baselines. In Section V, we analyze our proposed
method from several points of view. Finally, Section VI summarizes the proposed study and suggests future research directions.
II. Related Work

Learning a class-discriminative feature representation of EEG is
still challenging in both theory and practice. Numerous prior
studies have attempted to extract features from EEGs. In this
section, we briefly discuss linear methods and deep learning
models used for EEG signal representation.
A. Linear Models

Over the past decades, CSP [1] and its variants [11], [13] have
played an essential role in decoding MI. Blankertz et al. [1]
and Ang et al. [11] independently used a spatial filteringbased method for classifying MI. Ang et al. band-pass filtered
EEG data before applying CSP, thereby attempting to decode
EEG signals in a spatio-spectral manner. They named the

proposed method filter bank CSP (FBCSP). Furthermore,
Suk and Lee [13] also decoded MI by jointly optimizing
multi-spectral filters in a Bayesian framework and named
their method Bayesian spatio-spectral filter optimization
(BSSFO). Recently, Jin et al. proposed task-related channel
selection algorithms by using a correlation-based CSP training [31] and a bispectrum analysis [32]. Jin et al. [33] also
designed a selection algorithm of CSP-features based on the
Dempster-Shafer theory. Since conventional MI decoding
systems [1], [11], [13] were mostly developed based on subject-dependent manner that require considerable calibration
time, zero-calibration methods have been proposed in recent
years [27]. For instance, Lotte, Guan, and Ang [16] exploited
multi-resolution spectral decomposition (MR FBCSP) to
extract subject-neutral, i.e., subject-independent features.
Ray et al. [15] also extracted subject-independent features
using the FBCSP [11].
CCA is commonly utilized for detecting SSVEP [9] owing
to its practical ability to be implemented without the calibration stage. The standard CCA method deployed sinusoidal signals as reference signals and estimated canonical correlation
between the reference signals and input EEG signals to identify
an evoked frequency in SSVEP EEGs.
In addition, to characterize the sleep stage, entropy calculation-based approaches were frequently used. Sanders et al. [35]
classified the sleep stage using the spectral-temporal features of
EEGs learned from short-time Fourier transformation. Furthermore, Zheng and Lu [36] focused on identifying a driver's
mental fatigue during driving. They applied filter banks to
EEG signals to extract spectral information, and then transformed the filtered EEG signals to spectral space, i.e., estimated
PSD of filtered EEG signals. By doing so, Zheng and Lu effectively assessed the regression score of the driver's mental states
which were labeled using the PERCLOS index, a measure of
neurophysiological fatigue.
Earlier, Shoeb and Guttag [14] applied a machine learning
approach to extract and classify the spatio-spectral-temporal
features of epileptic seizure EEG signals. Specifically, they used
filter banks in a channel-wise manner to capture the spatiospectral information. Then, by encoding the temporal evolution
of extracted spatio-spectral feature vectors, they effectively constructed epileptic seizure EEG signal spatio-spectral-temporal
features and classified the seizure and non-seizure features utilizing a support vector machine (SVM). Recently, spectral features derived from a principal component analysis (PCA)
exhibited superior performance for seizure onset detection. In
particular, Lee et al. [37] band-pass filtered raw signals and calculated PSD. Then they applied PCA for the extraction of EEG
signal spectral features.
These practical linear model-based BCI methods [1], [9],
[11], [13], [35]-[37] have demonstrated credible performance.
However, these methods need to have certain prior neurophysiology knowledge [4], because their feature extraction stages are
specifically designed for each EEG paradigm. Conversely, our
method does not need to be specialized for different paradigms.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021