IEEE Computational Intelligence Magazine - May 2021 - 35

sep

f kST = C kST (x) = C k % C k - 1 % g % C 0 (x), (1)
sep
k

where C , C 0, and Fi % F j, respectively, denote the k-th separable convolution, the first temporal convolution, and a
function composition between arbitrary functions Fi and F j,
i.e., Fi % F j ($) = Fi (F j ($)). Thus, by extracting features
f 1ST, f 2ST, f, f NST, the MSNN effectively represents the spectraltemporal features from the multi-scale viewpoint, thereby automatically enhancing generalization. In addition, as all inputs are
zero-padded before each separable temporal convolution, the
output features have the same dimension for the channels and
timepoints, except for the feature map dimension.Thus, the k-th
spectral-temporal feature f kST now has the form [n c, nlT , Fk].
In summary, this spectral-temporal representation block in
our work extract spectral features of EEG signals at different
band-pass frequencies by convolving them with (1 # ( fs /2))
kernel. Specifically, based on Nyquist-Shannon sampling
theory, setting the kernel as (1 # ( fs /2)) allows our model to

C. Spatial Feature Representation Block

In the spatial feature representation block, a common spatial
convolution is used for feature extraction. In this block, the
kernel size is constrained to be equal to the number of EEG
channels; hence, a convolution with a kernel of (n c # 1) is used.
Additionally, by setting the kernel size to be the same as the

1 × (fs /2) Conv, lReLU, F0
1 × T1 SepConv, lReLU, F1
1 × T2 SepConv,
lReLU, F2

nc × 1 Conv,
lReLU, F1

1 × T3 SepConv,
lReLU, F3

nc × 1 Conv,
lReLU, F2

nc × 1 Conv, lReLU, F3

Spatial Feature
Representation Block

Spectral-Temporal Feature
Representation Block

An Input EEG Signal x

Concatenation

Global Average Pooling

Dense, Softmax, no
A Prediction of a User's
Intention or Condition y

capture spectral information at 2 Hz and above by multidimensional output filters, i.e., F0, [4], [22]. Then, the following
temporal separable convolutions with different kernel sizes
(1 # Tk), 6k = 1, f, N, convolve different spectral-temporal
features in a filter-independent manner. A brief graphical overview of this spectral-temporal feature representation block is
illustrated in Fig. 3(a).

Classification Block

efficient and explicit decoupling of the relationship between
the temporal and the feature map dimensions of the input features. This is accomplished by learning kernels independently
for each feature map. Thus, as in BCI literature, the separable
convolution [19] enables the system to learn temporal kernels
individually from the feature map dimensions (using a depthwise convolution [19]), and then optimally re-combine the feature maps (using a pointwise convolution [19]).
In this block, by setting a kernel size of (1 # Tk), where Tk
denotes the kernel size of the k-th temporal separable convolution, the k-th temporal separable convolutional layer represents
EEG signal features in the range of Tk /fsl sec, hence fsl /Tk Hz,
where fsl is a frequency property extracted at the first spectral
convolutional layer. Therefore, the spectral-temporal feature
representation layers can deal with different timepoints or frequency ranges by using various kernel sizes for the input
EEG data.
Additionally, each different layer that has a different kernel
size extracts features in different frequency and timepoint ranges. In other words, a spectral-temporal convolution layer with a
larger kernel represents longer-term temporal features, i.e., a lower-range of spectral features and vice versa. Then, the MSNN
exploits intermediate activations from each layer, thus learning
multi-scale feature representations.
In addition, a separable convolution [19] only operates convolutions in a cross-n T way, thus, the number of parameters is
small compared to a conventional convolution. For instance,
while a k-th separable temporal convolution has only
Tk + Fk - 1 $ Fk parameters, the conventional convolution with
the same size kernel has Tk $ Fk - 1 $ Fk parameters, where Fk
denotes the feature maps dimension of the k-th layer.
Furthermore, in this processing, as described above, the
MSNN uses its intermediate activations to exploit multi-scale
representations. In other words, the proposed network obtains
N numbers of spectral-temporal features f kST, k = 1, 2, f, N
like:

FIGURE 2 The proposed architectural framework of our multi-scale
neural network (MSNN). In the proposed network, first, an input EEG
x is temporally convolved (Conv) to expand the number of features,
where fs, F0, and lReLU denote the sampling frequency rate, the number of output filter maps of the first layer, and a leaky rectified linear
unit activation function respectively. Then, a set of temporal separable convolutions (SepConv) extracts spectral-temporal features (Tk
and Fk denote, respectively, the kernel size and output feature maps
of k-th temporal separable convolution, k = 1, 2, f, N, where N is the
number of temporal convolution layers). At the same time, a set of
spatial convolutions represents spatial features, where nc denotes the
number of acquired EEG channels. Then, the multi-scale features are
concatenated and fed into the global average pooling layer [39].
Finally, the dense layer (Dense) determines the class of an input EEG
by exploiting multi-scale features, where no denotes the number of
output nodes.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021