IEEE Robotics & Automation Magazine - December 2022 - 15

At each instant, the size of an ultrasound sample is 800 # 8,
where " 800 " represents the number of detection dots along
the detected ultrasonic echo direction, and " 8 " denotes the
number of transducers. Each transducer can detect information
from specific muscles, e.g., the first transducer mainly
detects the flexor carpi radialis, palmaris longus tendon, and
flexor pollicis longus muscles. In this study, we reshape the
size of the collected ultrasound to 10 # 80 # 8 so that the signal
on each layer (the size is 80 # 8) reflects the thickness of
the forearm muscles at different depths. Moreover, this
reshaping process significantly reduces parameter cardinality
in the following feature-extraction model.
The sEMG signal at time t is an eight-dimensional vector.
Considering that an sEMG is a typical time-sequence signal
whose information is contained within one period, the sEMG
sample corresponding to the force at time t is the sEMG signal
within the time-sliding window, resulting in the size of
each sEMG sample being 1 # 80 # 8.
Feature Extraction
Multichannel sEMG signals represent the activation information
of different muscles, coupled after biological electrical
signals are transmitted from deep muscles to the surface skin.
In this study, a multiview multikernel CNN (MMCNN)
model is proposed to extract deep features of the sEMG signals
from the time and frequency domains, respectively. The
MMCNN structure is depicted in Figure 6(a).
Considering submodel 2 in Figure 6(a), we use a CNN to
extract spatiotemporal information from the raw sEMG
signals. As the length of an sEMG sample is much larger than
its width (the length is 80, and the width is 8), the convolution
kernel of only one size can barely extract appropriate deep
features. To solve this problem, CNN block 1 involves five
convolution layers with different kernel sizes of 11 # 3, 21 # 3,
31 # 3, 41 # 3, and 51 # 3
to extract different timescale
deep features. The
sizes of all the polling layers
in this block are 5 # 1.
The five outputs of this
block have the same-sized
deep features. Accordingly,
in CNN block 2, five
convolution layers are
employed to further
extract features from the
aforementioned multitime-scale
deep features,
whose kernel sizes are 3 # 3, and all the pooling layers in this
block are 2 # 2, reducing the deep-features dimension. For
submodel 2 in Figure 6(a), spectrograms of the sEMG signals
are transformed from the corresponding time-domain sEMG
signals through a fast Fourier transform (FFT), and the sEMG
signal of each channel is transformed separately. Thus, the
size of the sEMG sample is 1 # 80 # 8. After the FFT, the
sEMG spectrograms are regarded as images, with the convolution
layers extracting deep features from the frequency
domain. Then, deep features of the sEMG signals from both
Estimating the covariance
matrices with regularization
also reduces the detection
of spurious correlations in
the training data.
Force
Force
Sample 1
Sample 2
Force
Sample 3
Force
Sample k
Force
(a)
sEMG
sEMG
Sample 1
sEMG
Sample 2
(b)
sEMG
Sample 3
sEMG
Sample k
Time (ms)
tt + 100
(c)
t + 200
t + 100 K
Figure 5. The alignment scheme of signals measured from multiple modalities.
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
15
Depth of the
Muscle
Ultrasonic
Sample 1
Ultrasonic
Sample 2
Ultrasonic
Sample 3
Ultrasonic
Sample k
Power of the Signal
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
