IEEE Robotics & Automation Magazine - December 2022 - 18

E(, )( )( )
ii ii+
u
xz pz px
specifies that the deep feature zi
is extracted from a random sample xi
in X .i F ()$
in this term
represents
a three-layer perceptron network, and ()$v is the activation
function.
Feature Fusion
We obtain the complementary and consistent information of
multiple modalities by optimizing latent representation
through (1) and (9). However, not all of the extracted deep
features are related to hand force estimation.
In this study, a self-attention layer is adopted to select
deep features from the optimized latent representation.
The latent representation of an sEMG sample is denoted
by
Xe ee [, ,, ],
ek
designated by Xu uu [, ,, ].uL k
= 12
-
f
XX Xse
T
=
u
T T
= 12 f and the one of the ultrasound is
T Their concatenated
T
feature is an L-dimensional vector, denoted by
[, ].
Considering that the deep features in the
latent representation have no definitive order, the positionencoding
layer is not necessary before they are input into
the self-attention layer. Therefore, output of the self-attention
layer is given by
Z = softmaxeoV ,
T
KQ
d
QW X
KW X
VW X
=
=
=
where W ,Q
W ,K
Q
K
V
and WV
s
,
,
,
s
s
(10)
are three weight matrices in the
self-attention layer to be trained. The output Z is the ultimate
latent representation of multiple modalities, which is
used for hand force estimation after being connected to
two fully connected layers, as shown in Figure 2. The
input and output dimensions of " FC1 " in Figure 2 are 128
and 32, respectively, and the input and output dimensions
of " FC2 " in Figure 2 are 32 and 1, respectively. The activation
functions of " FC1 " and " FC2 " are ELU and unit mapping,
respectively.
Through explaining each component of the proposed
OLR-SACNN model, it is evident that its overall loss function
comprises the CCEM (1), CCRM (9), and hand force
prediction losses (the mean-square error between the output
of the OLR-SACNN model and the ground-truth hand
force). In the model training stage, these three loss items
should be scaled to the same order of magnitude to balance
their effects.
Experiments
To verify the OLR-SACNN model's performance, we consider
experiments divided into three steps: 1) collect and process
multimodal data; 2) verify performance of the
MMCNN, MCNN, and OLR-SACNN models on hand force
estimation; and 3) test performance of the OLR-SACNN
model with data from clinical poststroke patients. It should
be noted that all the models are implemented on Ubuntu
18.04, utilizing an Nvidia GP102 Titan XP graphics card.
18 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
Model Validation
In this study, the performance of hand force estimation is
assessed based on the NMSE and the coefficient of determination
.R2
MMCNN Model Verification
The experiments challenge the proposed MMCNN model to
some representative traditional machine learning models like
support vector regression (SVR), k-nearest neighbor (KNN),
and gradient-boosting regression tree (GBRT). All the models
are tested on the NinaPro DB2 benchmark database, and
the sEMG samples from each subject are shuffled and divided
into two parts: 70% of samples are regarded as the training
set and 30% as the test set. Considering the traditional
machine learning models, four typical handcrafted sEMG
features are extracted, i.e., integrated electromyography
(EMG,) root-means square, zero crossing, and channel energy
percentage [14]. The dimension of each handcrafted feature
vector is 48 because each channel in one sEMG sample
(12 channels) can generate four features. The comparison
results illustrated in Figure 7(a) demonstrate that the forceestimation
performance of the MMCNN is better than SVR,
kNN, and GBRT. The average R2
based on the MMCNN
reaches 87.4%, and its average NMSE is 0.0634. The specific
force-estimation profiles of one subject are presented in Figure
7(b). The MMCNN's performance is also evaluated on
the self-collected data set. As illustrated in Figure 7(c), the
Data Collection
Performance of the MMCNN model is evaluated on the
NinaPro DB2 benchmark database as it contains the finger
output force and corresponding sEMG signals [20].
This database sampled sEMG signals of the biceps, finger
extensor, and triceps at 2,000 Hz using the Delsys Trigno
wireless system from 40 subjects. After being filtered
out by a third-order Butterworth filter to preserve the
20 + 50-Hz components, the sEMG samples are segmented
by a time-sliding window with an 80-ms length, creating
a 160 # 12 # 1 sEMG sample. Except for the sEMG
signal, the finger force signals are processed by an
eight-order low-pass Butterworth filter to preserve the
0 + 100-Hz component.
In addition, we built a multimodal data set containing
sEMG, ultrasound, and output force signals from 10
healthy subjects. This experiment was approved by the Ethics
Committee of the Institute of Automation, Chinese
Academy of Sciences on 5 April 2020 (IA-201931). According
to the experiment's protocol, the test subjects are
required to sit in front of the experiment table, wear the
sEMG and ultrasound armbands, put their forearms on the
table, and complete the hand grasp action according to the
guidance displayed on the laptop screen. During the task,
the subjects slowly increase their hand forces on the forcecollection
device, and then slowly release their hands from
the force-collection device. Each subject repeats this process
five times.
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
