IEEE Robotics & Automation Magazine - December 2022 - 20

0.004
0.008
0.012
0.016
0.2
0.4
0.6
0.8
1
(c)
0.016
0.012
0.008
0.004
124 8
The Number of Segment for Every Ultrasound Sample
10 16 20
(d)
0.0002
0.0004
0.0006
(e)
Figure 7. (Continued) (c) The NMSE and R2 of the sEMG-based force-estimation results of the MMCNN on the self-collected data
set. (d) The NMSE and R2 of the ultrasound-based hand-force-estimation results of traditional machine learning methods with the
different segment number n. (e) The NMSE and R2 of the ultrasound-based hand-force-estimation results of the MCNN.
NMSE of MMCNN is 0.0088, and the R2
is 66.71%.
of MMCNN
MCNN Model Verification
As there is no public database for estimating hand force
with A-mode ultrasound signals, the proposed MCNN
model's performance is verified on the self-collected data
set by challenging MCNN against traditional SVR, kNN,
and GBRT machine learning methods. The ultrasound
samples from each subject are shuffled and divided into
two parts: 70% of samples are regarded as the training set
and 30% as the test set. For traditional machine learning
methods, the following features are selected. For each
ultrasound sample, after being divided into n segments,
the mean value and standard deviation of each segment
are extracted as a feature, and thus, the feature dimension
is 16n for one ultrasound sample. In this experiment, n is
set to 16 through a trial-and-error method. The experiment
results in Figure 7(d) highlight that GBRT achieves
the best performance among the selected machine learning
models. However, according to Figure 7(e), the MCNN
outperforms all competing traditional machine learning
methods (NMSE of the MCNN is 0.00037, and R2
MCNN is 98.59%).
of the
20 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
OLR-SACNN Model Verification
First, we verify performance of the OLR-SACNN without the
CCEM and CCRM modules. In this case, deep features of
sEMG and ultrasound signals are directly concatenated without
being optimized. As illustrated in Figures 8(b), 7(c),
and 7(e), even without being optimized [F-concatenated in
Figure 8(b)], the OLR-SACNN based on multimodal information
still attains a better force-estimation result than the
single-modal methods (MMCNN and MCNN). R2
OLR-SACNN achieves 98.9%, increasing R2
of the
by 0.31 and
32.2% compared to the ultrasound- and sEMG-based
MMCNN, respectively; the NMSE is 0.00029, decreasing by
21.62% and 96.7%, respectively.
Second, the CCEM and CCRM modules are used in
the OLR-SACNN to optimize the latent representation.
Considering that the output dimension o of the CCEM
in (1) affects OLR-SACNN's performance, we evaluated
OLR-SACNN with different o values. The experiment
results in Figure 8(a) indicate that the OLR-SACNN can
reach the best performance for
o 4= , with the corresponding
force-estimation results on all subjects provided
in Figure 8(b). Compared to the case neglecting
the CCEM and CCRM modules [F-concatenated in
Figure 8(b)], the OLR-SACNN still improves the
0.97
0.98
0.99
1
1.2
1
0.8
0.6
0.4
0.2
124 8
The Number of Segment for Every Ultrasound Sample
10 16 20
GBRT KNN SVR
NMSE
NMSE
NMSE
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
AVG
0.00033
0.0003
0.00498
0.0049
0.0138
0.00054
0.00055
0.00027
0.00041
0.00039
0.00029
0.00026
0.00035
0.00037
R2
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
AVG
0.00355
0.0039
0.0112
0.00281
0.0035
0.0094
0.00234
0.003
0.0063
0.00193
0.0028
0.0069
0.00159
0.0023
0.0043
0.00181
0.0025
0.0055
R2
0.98594
0.98439
0.98564
0.97806
0.98758
0.98539
0.98062
0.9892
0.99195
0.98984
0.98586
0.7304
0.7397
0.3098
0.81517
0.7938
0.4899
0.86151
0.8173
0.5717
0.88857
0.8406
0.767
0.90593
0.8535
0.7368
0.92817
0.8876
0.8415
0.91357
0.8695
0.7903
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
AVG
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
AVG
0.0107
0.007
0.0137
0.0058
0.0073
0.0099
0.006
0.0114
0.0051
0.0114
0.0088
R2
0.5475
0.6357
0.6359
0.7683
0.6589
0.6416
0.702
0.5719
0.8424
0.6669
0.6671
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
