IEEE Robotics & Automation Magazine - December 2022 - 23

Table 2. The 12 recruited patients' information.
Patient Number
1
2
3
4
5
6
Age
48
63
49
63
64
61
Cause
Hemorrhagic
Hemorrhagic
Ischemic
Hemorrhagic
Ischemic
Hemorrhagic
Side
R
L
L
L
R
R
BS
5
4
5
6
4
5
Side: paralyzed side of the patient; L: left; R: right; BS: Brunnstrom stage of the patient.
Number
P7
P8
P9
P10
P11
P12
Age
78
22
43
60
47
73
Cause
Ischemic
Ischemic
Ischemic
Hemorrhagic
Ischemic
Hemorrhagic
Side
L
R
R
L
L
R
BS
4
4
6
5
6
5
Performance Test of OLR-SACNN by Clinical Data
This section applies the proposed OLR-SACNN model to
estimate poststroke patients' hand force. We collected clinical
data from poststroke patients at the China Rehabilitation
Research Center, and this clinical experiment was approved
by China Rehabilitation Research Center (2021-108-1) on
12 July 2021. The patient recruitment criteria are 1) between
18 and 80 years of age; 2) at Brunnstrom stage 4, 5, or 6 (the
hemiplegic hand has voluntary strengths); and 3) without
severe cognitive impairment. The exclusion criteria are
1) other neurological diseases, 2) orthopedic diseases, and
3) disability with learning how to use the hand-force-collection
device. Based on these criteria, we recruited 12 qualified
patients whose information is listed in Table 2. All patients
were given written and verbal information about this experiment,
and the signed informed consent statements were
received from the patients before the experiment. The experiment
protocol for poststroke patients is the same as for the
healthy subjects, and the collection process is illustrated in
Figure 3(b) and (c).
According to the experiment results presented in Figure
10, the average NMSE of the hand force estimation is
0.002349, and the average R2
is 0.79794. Compared to the
subjective assessment using the modified Ashworth scale,
this estimated hand force has the potential to provide a
quantitative strength rehabilitation assessment for poststroke
patients.
Discussion and Conclusion
This study proposed a multimodal fusion model named
OLR-SACNN to estimate hand force. An informationacquisition
system capable of simultaneously measuring
sEMG, ultrasound, and hand force was also presented.
The proposed OLR-SACNN attains satisfactory performance
on the self-collected data set, while compared to
the estimation methods based on a single modality
(sEMG or ultrasound), the NMSE of force estimation
decreases by 97.7% (sEMG) and 38.92% (ultrasound),
respectively. The proposed OLR-SACNN has also been
utilized to estimate the hand force of poststroke patients,
which shows its potential for robot-assisted rehabilitation
and quantitative assessment.
Our research shows that ultrasound-based estimation of
hand force outperforms sEMG based. Given that the human
muscle can be modeled as a unit-like spring, muscle force is
strongly related to the muscle's morphology. Therefore,
ultrasound has better performance, consistent with the muscle's
activity in nature. However, A-mode ultrasound has
been less prevalent than sEMG to estimate force in recent
years. For additional details, the reader is referred to the literature
related to ultrasound-based human-robot interaction
and the further broadening of its application domain.
This study also illustrated that force estimation based on
multiple modalities has a better accuracy than a single
0.003
0.0025
0.002
0.0015
0.001
0.0005
Subject
(a)
0.2
0.4
0.6
0.8
1
Subject
(b)
Figure 10. The NMSE and R2 of the estimated hand force by the
OLR-SACNN based on the poststroke patient's clinical data.
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
23
R2
NMSE
S1
S5
S11
S2
S8
S9
S3
S4
S7
S10
S6
S12
AVG
0.70757
0.72985
0.74473
0.76138
0.77329
0.77816
0.8298
0.84347
0.84921
0.85068
0.85261
0.8545
0.79794
S1
S5
S11
S2
S8
S9
S3
S4
S7
S10
S6
S12
AVG
0.001388
0.000986
0.00257
0.001295
0.00231
0.002367
0.001159
0.001066
0.00089
0.002412
0.000931
0.002453
0.002349

IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022