IEEE Robotics & Automation Magazine - December 2022 - 12

critical positions of the human muscles. Additionally,
an A-mode ultrasound can be designed as a lightweight
device with a thickness of only 0.6 mm [10], broadening its
practical applications.
However, the sEMG and A-mode ultrasound signals
describe partial attributes of the forearm muscles: the sEMG
records activation intensity of the muscle nerve, while the
ultrasound records changes of muscle morphology. Therefore,
a multimodal information fusion process can estimate
hand force, and thus we design a wearable system that can
synchronously collect the sEMG and A-mode ultrasound signals
to complete the hand-force-estimation task.
This study proposes a multimodal fusion model named
OLR-SACNN, whose architecture is shown in Figure 2. To
build this model, first, we extract some deep features of each
modality separately, which are then optimized, employing
two custom modules. Finally, the optimized latent representation
is fused with the self-attention module and then used for
hand force estimation. This study is divided into the following
three stages:
1) A multimodal data-acquisition system aimed at hand force
estimation is designed.
2) Deep learning models are studied to estimate hand force
with either the single sEMG or single A-mode ultrasound.
3) A multimodal fusion model is further investigated to
obtain a more accurate estimation of hand force with
experimental verification.
Related Work
Learning Methods on a Single Modality
In recent years, sEMG signals have been studied for hand gesture
recognition, hand (or wrist) force estimation [11], [12],
and knee-ankle-joint motion regression [13]. Traditional
machine learning methods have analyzed sEMGs by extracting
handcrafted time- and frequency-domain features to
complete sEMG-based recognition tasks [14], [15]. However,
selecting these handcrafted features depends highly on the
practitioner's experience.
Recent studies have confirmed that deep learning
models such as long short-term memory, CNNs, and
transformers outperform traditional machine learning
methods. Additionally, the automatic feature-extraction
abilities of these deep learning models afford sEMG signals
to be easily preprocessed by some digital filters [16] without
involving a complicated, handcrafted feature-extraction
process.
Nevertheless, A-mode ultrasound signals have been
rarely studied for hand force estimation. One exception is
the work in [17], where the absolute estimation error
reached 1.04%, but in this method, preprocessing the
A-mode ultrasound is complicated, including time-gain
compensation, digital filtering, envelope detection, log
compression, and signal segmentations. After that, the
mean and standard deviation are selected as features for the
estimation task [18]. To the best of our knowledge, a deep
learning method has never been applied to the A-mode
ultrasound for hand force estimation, which is the research
goal of this study.
Multimodal Fusion Methods
In the multimodal fusion scenario, each modality describes
the object from a partial view, while multiple modalities
involve multiple views. Learning a latent representation that
preserves the complementary information and extracts the
consistent information from these different views is challenging.
Given that the learned latent representation should
reach a satisfactory accuracy in realizing downstream tasks,
x(1)
1
x(1)
1
x(1)
2
x(1)
N-1
x(1)
N
Raw
sEMG
Spectral
sEMG
x(v)
1
x(v)
2
f(2)(.)
x(v)
N-1
x(v)
N
Raw Ultrasonic
CCRM
x(v)
N-1
x(v)
N
x(v)
1
x(v)
2
f(1)(.)
CCRM
x(1)
2
x(1)
N-1
x(1)
N
SelfAttention
CCEM
Figure
2. A framework of the proposed multimodal fusion model OLR-SACNN. FFT: fast Fourier transform LR-EMG: latent
representation of sEMG; LR-Ul: latent representation of ultrasound.
12 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
FFT
Latent
Representation
Fused Multimodal Features
FC1
FC2
LR-EMG
LR-UI
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
