IEEE Computational Intelligence Magazine - May 2021 - 32

proposed method, we conducted experiments on various paradigms of active/passive BCI datasets. Our experimental results
demonstrated that the proposed method achieved performance
improvements when judged against comparable state-of-the-art
methods. Additionally, we analyzed the proposed method using
different techniques, such as PSD curves and relevance score
inspection to validate the multi-scale EEG signal information
capturing ability, activation pattern maps for investigating the
learned spatial filters, and t-SNE plotting for visualizing represented features. Finally, we also demonstrated our method's
application to real-world problems. Based on our experimental
results and analyses, we believe that the proposed multi-scale
neural network can be useful for various BCI paradigms, as a
starting model or as a backbone network in any new BCI
experiments.

B

I. Introduction

rain-computer interface (BCI) [1], [2] is an emerging
technology that enables a communication pathway
between a user and an external device (e.g., a computer) through the acquisition and analysis of brain signals. These signals are then translated into commands that are
understood by a device, such as a computer. Owing to its
practicality, electroencephalogram (EEG)-based non-invasive
BCIs are widely used [1]-[5]. Earlier, Aricò et al. [6] categorized user-centered BCIs into two types, active/reactive and
passive BCIs. In this paper, our focus is not only on active
BCIs but also on passive BCIs. Generally, two types of brain
signals such as evoked and spontaneous EEG are primarily considered for active/reactive BCIs [7]. Evoked BCIs exploit
unintentional electrical potentials reacting to external or
internal stimuli. Examples of evoked BCIs include steadystate visually evoked potentials (SSVEP) [8], [9] and eventrelated potentials such as P300 [8], [10]. Additionally,
spontaneous BCIs use an internal cognitive process such as
event related desynchronization and event related synchronization (ERD/ERS) in sensorimotor rhythms, e.g., motor
imagery (MI) [4], [11] induced by imagining movements in
addition to physical movement. Well-known examples of
passive BCIs include the use of sleep/drowsy EEG signals for

µ V 2/Hz (dB)

158

135

5 10 15 20 25 30 35 40 5 10 15 20 25 30 35 40
Frequency (Hz)
Frequency (Hz)

FIGURE 1 Power spectral density (PSD) curves of two different subjects' MI EEG samples. The solid red line denotes the mean PSD and
the shaded region exhibits the standard deviation for all trials. Clearly,
these two different subjects show quite different PSD patterns for the
same paradigm. Note that two PSD plots share the y-axis label.

32

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021

sleep stage classification or identifying mental fatigue to alert
a driver of a dangerous situation and seizure EEG patterns
for onset detection to provide the patient with a warning of
a potential seizure.
Generally, machine learning-based BCIs consist of five main
processing stages [4]: (i) an EEG signal acquisition phase based
on each paradigm, (ii) signal preprocessing (e.g., channel selection and band-pass filtering), (iii) feature representation learning, (iv) classifier learning, and finally (v) a feedback stage.
Basically, most machine learning-based BCI methods follow
these processes, however, these methods need specific modification to classify a user's intention/condition for each different paradigm [4]. In other words, machine learning-based
methods need to have prior knowledge of different EEG paradigms [1], [4], [8], [9], [12]. Therefore, conventional machine
learning-based BCIs have discovered EEG representations
through extremely specialized approaches, e.g., a common
spatial pattern (CSP) [1] or its variants [11], [13] for MI signals and a canonical correlation analysis (CCA) [9] for SSVEP
signals decoding.
While hand-crafted feature representation learning has
a pivotal role in a conventional machine learning framework [1], [9], [14]-[16], deep learning-based representation has had remarkable results in the BCI community [3],
[4], [7], [17]. These deep learning-based methods have
integrated a feature extraction step with a classifier learning step such that those steps are jointly optimized, thereby
improving performance. Among various deep learning
methods, convolutional neural networks (CNNs) have the
advantage [4], [18], [19] of maintaining the structural and
configurational information in the original data. In this
respect, developing a novel CNN architecture for EEG
signal representation has taken a center stage in the BCI
studies [3], [18], [20]-[27].
However, some challenges still remain. First, existing CNNbased methods [3], [18], [20], [21], [25], [26], [28] are mostly
comprised of stacked convolutional layers. In other words,
those existing methods extract features sequentially. But, ignoring multiple ranges of spectral-temporal features can cause a
critical problem because EEG signal features for different subjects [29], paradigms [4], and types [6] are found in diverse
ranges. For example, Fig. 1 depicts two different subjects' MI
EEG power spectral density (PSD) curves. Clearly, these two
plots have different distributions from each other even though
these PSDs are estimated by the same task. Therefore, it is
important to capture multi-scale spectral information in EEGs
for general use in BCI, i.e., a generic method applicable to various types of BCIs.
In addition, those stacked CNN-based methods [3], [18],
[25], [26] have numerous trainable parameters, thus requiring
large amounts of training samples, whereas BCIs generally
acquire a limited number of EEG trials [29]. Therefore, generalizing conventional stacked CNN-based methods in BCI is
quite difficult because deep learning is a data-hungry problem,
i.e., rarely generalized with a lack of data.
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
