IEEE Robotics & Automation Magazine - December 2022 - 17

the time and frequency domains are obtained, concatenated,
and fused via a convolution layer with a kernel size of 1 # 1.
Then, deeper convolution layers are applied to obtain more
complex information about the sEMG signals. It is noted that
all the activation functions in the convolution layers are Exponential
Linear Unit (ELU).
The ultrasound signals from eight channels are also coupled,
similarly to the sEMG. Precisely, one transducer
captures its ultrasonic echo or the echoes from other transducers.
In addition, there is a complex coordination mechanism
among the forearm muscles when the hand exerts
force, imposing a complicated mapping between muscle
state and hand force. Therefore, we fully, deeply exploit the
learning of automatic feature-extraction ability to analyze
ultrasound signals and conclude that the feature extraction
of such signals suffers from the same problems as sEMGs:
the length is much larger than the width. Hence, we propose
a multikernel CNN (MCNN) model for the ultrasound,
which is depicted in Figure 6(b). An MCNN has
the same structure as an MMCNN, except that submodel 2
is omitted because the ultrasound sample is an instantaneous
signal.
Optimization of Latent Representation
After deep features of the sEMG and ultrasound signals
are extracted, they need to be further optimized to achieve
better performance on hand force estimation. To this end,
the following CCEM is proposed, whose optimization
objective is
ii
max
12
,, ,UV
1 trace Uf xf xV11 22
T T
subject to U ` fx fx rI UI ,## (2)
j
T 1 T
m
11 11 64 64
V ` m 22 22 64 64
T T
i
() ()+= oo
j
T 1 fx fx rI VI
T
uf () () ,,xf xv12 2
j =0f ijor
!
() ()+= oo
## (3)
,
(4)
where m is the number of samples in a batch during model
training, and o is the dimension of the optimized deep feature
generated by the CCEM. x Rm
input sEMG sample, and x Rm## #
input ultrasound sample. f () Rm#64
1 !
2 !
1 $ !
2 $ !
#
## #1808
10 80 8
represents the model utilized for obtaining
respectively. Both U R o64
$
and (),f2
$
!
#
!
are the matrices to be optimized. ui
ii
))
1 !
2 !
mal parameter set (, ),12
and V, respectively:
and ()f2
$
the output deep features of ()f1
$
are projected as H Rmo
oo
and H Rmo
#
where ()p $
represents the
symbolizes the
denotes the model
used for extracting the deep features of sEMG signals, and
f () Rm 64
deep features of the ultrasound signals. 1i and 2i are the
parameters of ()f1
and V R o64
#
column of U, and vj
is the ith
is the jth column of V. To find the opti#
maxmax
JS
pz x
by U
HH ,.11 12 22rr=m
=-
HH H
m
oo
pz xT xzi
ii
ii
;
E
IX Zpzx px pz px
E
(, )( () () () ())
[( (, ))]
v Fx z
ii+; <
=
+;
()
()
(, )
+
(, )( )( )
xz pz px
ii ii
xz pz xpxi i
u [( (( ,ii))].
ii ii i
(, )( )( )
ii +;
ii i
1
uu
u
maxlog
log
11H11 (5)## in this term is extracted from the corresponding ,xi
u
In (9), E(, )( )( )
xz pz xp x
v Fx z
(9)
ii + ;ii i indicates that the deep feature zi
while
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
17
m
(( )( )),
(1)
The covariance 11R of
their cross covariance 12
R =t ij
m 1
-
i
H ,1
r
covariance 22R of
rr + #jo o
R are defined as
,, ,,
ij !
1 HH rI ^h " 12,
T
H ,2
r
and
(6)
where r 02 is a regularization parameter. Estimating the
covariance matrices with regularization also reduces the
detection of spurious correlations in the training data. By
defining
T
_RR R-tt
t
12
11
corr (, )HH
12
deep networks ()f1
$
(/ )( /)
12 22
and ()f2
$
12 =
12 , the training objective of the
is to make the correlation
as large as possible:
(, )( ).
corr HH trace TT
T
2
1
(7)
The CCEM would lead to complementary components
of the multiple modalities being discarded while their common
components are extracted. Generally, the autoencoder
can preserve the complementary information of multiple
modalities by reconstructing input sEMG and ultrasonic
signals using the extracted deep features. However, the autoencoder
would extract the deep noise features, which can
decrease the reconstruction error but are useless for estimating
hand force. In this view, the CCRM is proposed in the
following paragraph, which employs the mutual information
to extract essential information from the sEMG and
ultrasonic signals.
Let the original data set (the sEMG or ultrasonic signal) be
Xi 12! "
denoted by ,, ,, and let xX
and zZ
i
ii
of xi. The mutual information of Xi
IX Zp xz
(, )( ,)
ii
=
=
##
ii
ii
within this set. The extracted deep feature set from Xi
denoted by ,Zi
is given by
dxdz
KL Pz xx pz px
ii ii i
;<
uu
pi
required that the mutual information (, )IX Zii
to make the deep feature zi
Pz xp xii i and () (). As the Kullback-Leibler diver;
u
u
pz
px
ii
(8)
(( )( )( )( )),
log
px pz
px z
() ()
(, )
ii
ii
represents the probability density function. It is
be maximized
describe the corresponding xi
and zi
more exactly, but not the other original samples. Equation (8)
indicates that the mutual information of xi
is equal to
maximize the distance between the distribution of
() ()
gence has no upper bound while being maximized, which
may lead to the failure of model training, the Jensen-Shannon
divergence is utilized to maximize the mutual information.
Hence, (8) is transformed into
! be one sample
is
! is the corresponding deep feature
and Zi
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
