IEEE Computational Intelligence Magazine - May 2021 - 36

number of electrode channels, similar to many existing deep
learning-based BCI methods [3], [4], [18], the proposed
MSNN extracts spatial information from the original EEG
acquisition channel distributions of multi-scale spectral temporal features. Then, the MSNN can obtain neurophysiologically
plausible information from the input data distribution.
Furthermore, the spatial feature representation can be
applied unrestrictedly, thus in the proposed method, we add
this block after every extracted spectral-temporal feature
f kST, k = 1, 2, f, N like,
f kSST = S k (f kST) = S k % C kST (x), (2)

	

where S k denotes the k-th spatial convolution and f kSST are
spatio-spectral-temporal features estimated by the S k and C ST
k .
We use valid paddings for every spatial convolution, thus the
k-th spatio-spectral-temporal feature f kSST has the form
[1, nlT , Fk]. By setting the number of spatial convolutions to
be identical to the number of spectral-temporal convolutions,
unlike many previous research using deep learning for BCI
[3], [4], [17], [18], [20], [24], we extract spatial features of
each range from spectral-temporal features. In other words,
unlike many previous stacked CNNs, the proposed architecture uses every intermediate activated feature set to exploit
spatial information, thereby creating the capability to extract
various ranges of EEG features at multi-scale. Refer to Fig.
3(b) for the graphical overview of the spatial feature representation block.
D. Classification Block

For classifier learning, because we have N numbers of different
(or same when F1 = F2 = g = FN ) size of spatio-spectral-temporal features f kSST, k = 1, 2, f, N, the classifier in the proposed
method has to concatenate the features in the feature map
SST
dimension.Thus, the concatenated feature f concat
is represented as:
N

N

i=1

i=1

SST
f concat
= < f iSST = < S i % C ST
i (x), (3)

	

where < denotes the concatenation operation.
For the classifier network, let us assume that the number of
output classification nodes is denoted by n o and we use a single linear mapping layer. Then, we need to train the large
number of R Ni = 1 n o $ nlT $ Fi parameters (note that we disregard
SST
the bias term for simplicity) because f concat
has the form
N
[1, nlT , R i = 1 Fi], and it would still require a large number of
training samples. Therefore, after representing the input EEG
data to multi-scale spatio-spectral-temporal features, the proposed MSNN has one extra operation for reducing the trainable weights. Unlike the existing deep learning-based BCI
methods [3], [4], [17], [18], [20], [23], [24], global average
pooling (GAP), which is widely used in the computer vision
field [39], is performed.
The GAP layer [39], a type of pooling layer, averages nodes
from each feature map, thus eliminating the requirement for
any window size or stride. By applying GAP, our proposed
MSNN efficiently extracts significant features. From the BCI
literature, the GAP layer can be understood to be a method
that can emphasize an important frequency range and its surrounding area for each feature map dimension. Thus, for the
extracted multi-scale features in the MSNN, the GAP layer
[39] stresses the crucial spectral-temporal part resulting in concise information for the final decision making.
Additionally, the GAP layer [39] significantly reduces the
number of classifier parameters used in the proposed MSNN.
Specifically, after the GAP layer G ($), the extracted feature is
reduced to the form [1, 1, R Ni = 1 Fi], whereas the feature without
GAP has the form [1, nlT , R Ni = 1 Fi]. Therefore, we drastically
suppress the trainable parameters in the classifier from
nlT $ n o $ R Ni = 1 Fi to n o $ R Ni = 1 Fi.
Then, the MSNN prediction, yt , for the input EEG data, x,
is as follows:

	

SST
h + b oh
yt = softmax ^W 


IEEE Computational Intelligence Magazine - May 2021
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
Contents


IEEE Computational Intelligence Magazine - May 2021 - Cover1

IEEE Computational Intelligence Magazine - May 2021 - Cover2

IEEE Computational Intelligence Magazine - May 2021 - Contents

IEEE Computational Intelligence Magazine - May 2021 - 2

IEEE Computational Intelligence Magazine - May 2021 - 3

IEEE Computational Intelligence Magazine - May 2021 - 4

IEEE Computational Intelligence Magazine - May 2021 - 5

IEEE Computational Intelligence Magazine - May 2021 - 6

IEEE Computational Intelligence Magazine - May 2021 - 7

IEEE Computational Intelligence Magazine - May 2021 - 8

IEEE Computational Intelligence Magazine - May 2021 - 9

IEEE Computational Intelligence Magazine - May 2021 - 10

IEEE Computational Intelligence Magazine - May 2021 - 11

IEEE Computational Intelligence Magazine - May 2021 - 12

IEEE Computational Intelligence Magazine - May 2021 - 13

IEEE Computational Intelligence Magazine - May 2021 - 14

IEEE Computational Intelligence Magazine - May 2021 - 15

IEEE Computational Intelligence Magazine - May 2021 - 16

IEEE Computational Intelligence Magazine - May 2021 - 17

IEEE Computational Intelligence Magazine - May 2021 - 18

IEEE Computational Intelligence Magazine - May 2021 - 19

IEEE Computational Intelligence Magazine - May 2021 - 20

IEEE Computational Intelligence Magazine - May 2021 - 21

IEEE Computational Intelligence Magazine - May 2021 - 22

IEEE Computational Intelligence Magazine - May 2021 - 23

IEEE Computational Intelligence Magazine - May 2021 - 24

IEEE Computational Intelligence Magazine - May 2021 - 25

IEEE Computational Intelligence Magazine - May 2021 - 26

IEEE Computational Intelligence Magazine - May 2021 - 27

IEEE Computational Intelligence Magazine - May 2021 - 28

IEEE Computational Intelligence Magazine - May 2021 - 29

IEEE Computational Intelligence Magazine - May 2021 - 30

IEEE Computational Intelligence Magazine - May 2021 - 31

IEEE Computational Intelligence Magazine - May 2021 - 32

IEEE Computational Intelligence Magazine - May 2021 - 33

IEEE Computational Intelligence Magazine - May 2021 - 34

IEEE Computational Intelligence Magazine - May 2021 - 35

IEEE Computational Intelligence Magazine - May 2021 - 36

IEEE Computational Intelligence Magazine - May 2021 - 37

IEEE Computational Intelligence Magazine - May 2021 - 38

IEEE Computational Intelligence Magazine - May 2021 - 39

IEEE Computational Intelligence Magazine - May 2021 - 40

IEEE Computational Intelligence Magazine - May 2021 - 41

IEEE Computational Intelligence Magazine - May 2021 - 42

IEEE Computational Intelligence Magazine - May 2021 - 43

IEEE Computational Intelligence Magazine - May 2021 - 44

IEEE Computational Intelligence Magazine - May 2021 - 45

IEEE Computational Intelligence Magazine - May 2021 - 46

IEEE Computational Intelligence Magazine - May 2021 - 47

IEEE Computational Intelligence Magazine - May 2021 - 48

IEEE Computational Intelligence Magazine - May 2021 - 49

IEEE Computational Intelligence Magazine - May 2021 - 50

IEEE Computational Intelligence Magazine - May 2021 - 51

IEEE Computational Intelligence Magazine - May 2021 - 52

IEEE Computational Intelligence Magazine - May 2021 - 53

IEEE Computational Intelligence Magazine - May 2021 - 54

IEEE Computational Intelligence Magazine - May 2021 - 55

IEEE Computational Intelligence Magazine - May 2021 - 56

IEEE Computational Intelligence Magazine - May 2021 - 57

IEEE Computational Intelligence Magazine - May 2021 - 58

IEEE Computational Intelligence Magazine - May 2021 - 59

IEEE Computational Intelligence Magazine - May 2021 - 60

IEEE Computational Intelligence Magazine - May 2021 - 61

IEEE Computational Intelligence Magazine - May 2021 - 62

IEEE Computational Intelligence Magazine - May 2021 - 63

IEEE Computational Intelligence Magazine - May 2021 - 64

IEEE Computational Intelligence Magazine - May 2021 - 65

IEEE Computational Intelligence Magazine - May 2021 - 66

IEEE Computational Intelligence Magazine - May 2021 - 67

IEEE Computational Intelligence Magazine - May 2021 - 68

IEEE Computational Intelligence Magazine - May 2021 - 69

IEEE Computational Intelligence Magazine - May 2021 - 70

IEEE Computational Intelligence Magazine - May 2021 - 71

IEEE Computational Intelligence Magazine - May 2021 - 72

IEEE Computational Intelligence Magazine - May 2021 - 73

IEEE Computational Intelligence Magazine - May 2021 - 74

IEEE Computational Intelligence Magazine - May 2021 - 75

IEEE Computational Intelligence Magazine - May 2021 - 76

IEEE Computational Intelligence Magazine - May 2021 - 77

IEEE Computational Intelligence Magazine - May 2021 - 78

IEEE Computational Intelligence Magazine - May 2021 - 79

IEEE Computational Intelligence Magazine - May 2021 - 80

IEEE Computational Intelligence Magazine - May 2021 - 81

IEEE Computational Intelligence Magazine - May 2021 - 82

IEEE Computational Intelligence Magazine - May 2021 - 83

IEEE Computational Intelligence Magazine - May 2021 - 84

IEEE Computational Intelligence Magazine - May 2021 - 85

IEEE Computational Intelligence Magazine - May 2021 - 86

IEEE Computational Intelligence Magazine - May 2021 - 87

IEEE Computational Intelligence Magazine - May 2021 - 88

IEEE Computational Intelligence Magazine - May 2021 - 89

IEEE Computational Intelligence Magazine - May 2021 - 90

IEEE Computational Intelligence Magazine - May 2021 - 91

IEEE Computational Intelligence Magazine - May 2021 - 92

IEEE Computational Intelligence Magazine - May 2021 - 93

IEEE Computational Intelligence Magazine - May 2021 - 94

IEEE Computational Intelligence Magazine - May 2021 - 95

IEEE Computational Intelligence Magazine - May 2021 - 96

IEEE Computational Intelligence Magazine - May 2021 - 97

IEEE Computational Intelligence Magazine - May 2021 - 98

IEEE Computational Intelligence Magazine - May 2021 - 99

IEEE Computational Intelligence Magazine - May 2021 - 100

IEEE Computational Intelligence Magazine - May 2021 - Cover3

IEEE Computational Intelligence Magazine - May 2021 - Cover4


https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805

https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12

https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12


https://www.nxtbookmedia.com