IEEE Robotics & Automation Magazine - December 2022 - 21

performance as the NMSE decreases by 11.54%, and R2
increases by 0.24%.
Third, as discussed in the " Data Processing " section, the
CCRM adopts the mutual information mechanism rather
than the autoencoder to optimize the latent representation. To
verify that the OLR-SACNN adopting the mutual information
mechanism can obtain a better result, we perform comparative
experiments against the mutual information
mechanism, revealing that the NMSE decreases by 0.27%, as
shown in Figure 8(b).
Fourth, we test estimation frequency of the OLR-SACNN.
For each multimodal sample, the average processing time is
roughly 0.3 ms. Considering that the
time consumed for data collection is
10 ms (the sampling rate is 100 Hz),
the total time required is 10.3 ms.
Therefore, the OLR-SACNN's estimation
frequency can reach 99.1 Hz.
Thus, it is easy to see that the data
sampling rate mainly limits the estimation
frequency, and it can be dramatically
improved if an advanced
data-collection device is employed.
Analysis of OLR-SACNN
We extract and compare deep features
of the sEMG and ultrasound signals
to analyze how the OLR-SACNN balances
information from multiple
modalities. As illustrated in Figure
9(a), the solid regression line is
obtained by fitting the scatters (each
scatter represents one sample) with
the least-squares method. The slope
of the regression line is 3.8427, indicating
that in OLR-SACNN's latent
representation, ultrasound plays a
more critical role. This result is consistent
with the previous experiment's
results, proving that force
estimation based on an ultrasound
signal is better than an sEMG one.
To analyze the function of the
CCEM, the correlation between the
deep feature of the sEMG and the
ultrasound is calculated. Specifically,
both deep features of the sEMG and
the ultrasound ( " latent representation
of sEMG " and " latent representation
of ultrasound " in Figure 2) are
processed by the principal component
analysis method to generate
simplified features with a dimension
of 16. Then the correlation matrix of
these two 16-dimensional features is
calculated, considering whether the
CCEM is adopted or not. As presented in Figure 9(c), the
right correlation matrix (adopting CCEM) has larger values
than the left (not adopting CCEM), indicating that the
CCEM module prompts the OLR-SACNN to extract more
relevant deep features from different modalities.
To analyze whether or not the CCRM can guide the
OLR-SACNN to extract essential features from the samples
rather than noise, the correlation between the input samples
and their corresponding deep features is determined
by calculating their distance. In Figure 9(c), the horizontal
axis represents the distance between the extracted deep feature
and the input sample when the CCRM module is
0.0005
0.001
0.0015
1248 16 32 64
0.92
0.94
0.96
0.98
1
1 248 16
(a)
0.0001
0.0002
0.0003
0.0004
0.0005
0.0006
0.0007
0.0008
0.0009
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 AVG
0.975
0.98
0.985
0.99
0.995
1
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 AVG
OLR-SACNN F-Concatenation AE-SACNN
(b)
Figure 8. Experiment results of the hand force estimation based on the OLR-SACNN
with different parameters and structures. (a) The NMSE and R2 of the force estimation
based on the OLR-SACNN with different values of o in (1). (b) The NMSE and R2 of the
hand force estimation based on based on the OLR-SACNN with different structures:
" OLR-SACNN " represents the case when the deep feature is optimized by the CCEM and
CCRM modules, " F-Concatenation " denotes the case without feature optimization, and
" AE-SACNN " signifies the case when the deep feature is optimized by the autoencoder.
AVG: average.
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
21
32
64
R2
NMSE
R2
0.9871
0.9887
0.989
0.985
0.9886
0.9819
0.9857
0.9876
0.9894
0.9885
0.9872
0.9925
0.9908
0.99
0.9893
0.9858
0.9923
0.9915
0.9904
0.989
0.989
0.9887
0.9917
0.9914
0.9889
0.9923
0.9947
0.9941
0.9912
0.9908
0.9929
0.993
0.9913
0.000207
0.000179
0.000268
0.000289
0.000161
0.000278
0.000217
0.000203
0.000171
0.000284
0.000226
0.000166
0.000169
0.000304
0.000219
0.000414
0.000361
0.000267
0.000293
0.000232
0.000203
0.000276
0.000328
0.00029
0.000377
0.000307
0.000563
0.000324
0.000199
0.000503
0.000287
0.000296
0.000190
NMSE
0.9771
0.9747
0.00065
0.00072
0.9915
0.9616
0.9534
0.9632
0.9623
0.00024
0.00118
0.00133
0.00105
0.00107
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
