IEEE Systems, Man and Cybernetics Magazine - October 2023 - 49

utilized CNN as a deep learning framework for identifying
the gesture recognition needs of AUS signals.
Dynamic Gesture Recognition Strategy
RNN Strategy of Dynamic Gesture Recognition
The structure of the whole algorithm is shown in Figure 2.
The LSTM part is a double-layer LSTM NN with 100 hidden
layers. The output result is connected with the full
connection layer with 128 hidden layers and finally sent to
the second full connection layer with the set number of
categories. The dropout part sets the parameter to 0.5.
After prepossessing the raw signal, we can get a 3D
matrix with the dimension of total frames * channels *
sampling points, which is called the original data matrix.
Because the core of the algorithm is the deep learning
framework of LSTM, we need to recast the
dimension of the original data matrix into
a 3D matrix of action times * frames per
action * (channels * sampling points). This
matrix is called the dynamic action
matrix. Through the change of the matrix
dimension and the use of LSTM, we take
each frame as the object of feature extraction.
We take the whole process of acting
as a whole and realize the interframe correlation
(IFC) in the process of a single
action. The realization of IFC makes up
for the lack of time information about the
AUS signal so that
dynamic actions.
Input
LSTM Network
Result
Figure 2. The RNN architecture.
it can recognize
CNN Strategy of Dynamic Gesture Recognition
CNN is often used to process image data
recognition, with three channels (red,
green, and blue) for most pictures. For the
need for dynamic gesture recognition, we
first compare the number of frames in the
whole motion to the number of channels
in the picture. Therefore, a strategy for
CNN dynamic gesture recognition is
equivalent to learning a picture with 40
channels. With this method, we can solve
the problem that dynamic gestures cannot
be recognized due to the lack of time
information in the AUS signal. The data
structure of CNN is similar to that of RNN. We use one
frame of data from four channels as a sample and then
transform each frame into a shape of channels * sampling
points (40 × 4 × 960) for each sample to feed into the
network for training. The network structure consists of
five layers, as shown in Figure 3.
The first layer is the input layer with input shape
Input
Result
Conv.
2D Convolution
Down. n ×1/n Downsampling
Figure 3. The CNN architecture.
Table 1. The CNN's architecture.
In
ConvBlock 1 40
Pooling 1
(batch size: 40, 4, 960); the second layer is the convolution
layer, which contains 160 convolution kernels of 1 × 1. The
stride and padding are both 1 for extracting local
40
ConvBlock 2 160
Pooling 2
160 160
32
40
(1, 51)
(1, 5)
(1, 23)
(1, 5)
Soft.
FC
Fully Connected
Softmax
features; after the convolution operation, batch normalization
is performed, and the rectified linear unit (ReLU)
is utilized as the activation function for extracting the
abstract features of the AUS signal. The next two fully
connected layers contain 4,096 and class number units,
respectively. To avoid overfitting and speed up the processing,
the batch normalization operation is also used
after each fully connected layer. The activation function is
ReLU with a random deactivation rate of 0.5. The softmax
was used as the classification activation function, and the
optimizer was Adam; the initial learning rate was 0.00001.
The specific parameter settings are shown in Table 1.
Dataset
In this article, the experimental dataset for dynamic gesture
recognition is composed of AUS signals. Four
ct-1
ht-1
ct
ct+1
LSTM
LSTM
ht
FC FC
ht+1
Out kernel_size Padding Stride
160
(1, 5)
(1, 5)
(1, 1)
(1, 5)
(1, 1)
(1, 5)
October 2023 IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE 49
Conv.
xt+1
Down. 5
Conv.
Down. 5
FC
FC
FC
Soft.
xt
yt+1
yt
IEEE Systems, Man and Cybernetics Magazine - October 2023

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - October 2023
