IEEE Robotics & Automation Magazine - December 2022 - 13

1
2
7
3
4
5
6
(a)
(b)
(c)
Figure 3. The proposed multimodal data-acquisition system. (a) A display of the data-collection process from one health subject, with
key components of the system: 1) software for the sEMG, force, and ultrasound collection; 2) a force-collection device; 3) an eightchannel
stabilizer; 4) an NI data-acquisition unit; 5) sEMG armband; 6) ultrasound armband; and 7) ultrasound instrument. (b) A
display of the scenario that one patient was instructed about the force-estimation study. (c) A display of one snapshot during the
acquisition system in operation.
this study employs a canonical correlation analysis to learn
the correlated components of different deep features from
multiple modalities.
To avoid discarding the complementary information of
different modalities while extracting consistent information,
it is necessary to build a function that measures information
loss between the latent representation and its
corresponding input signals, which is then used to preserve
each modality's complementary components. To this end,
Wang et al. utilized an autoencoder to restore the input
with its extracted deep features [19]. However, minimizing
the restored error in the autoencoder may lead to the latent
representation reserving noise information, which is helpful
for reconstruction but unnecessary for characterizing
the input signal. This work utilizes mutual information to
retain the complementary components from multiple
modalities to solve this problem.
Materials and Methods
Architecture of the Multimodal System
The overall architecture of the multimodal data-acquisition
platform comprises eight parts: one sEMG armband
(OYMotion Technologies, China), one ultrasound instrument
(Shantou Ultrasonic Electronics Company), one dataacquisition
unit [NI USB-6008, National Instruments (NI),
USA], one eight-channel stabilizer (Junyi Technology,
China), one customized hand-force-collection device, one
customized ultrasound armband, and one customized datacollection
software tool. The data-acquisition system and
its components are shown in Figure 3(a). The sEMG armband,
ultrasound, and force-measurement devices are connected
to the computer through a Bluetooth adapter,
Ethernet, and serial port. The sEMG and ultrasound armbands
have eight channels and are worn on the forearm.
Moreover, the National Instruments-Data AcQuisition.
Unit and the stabilizer collect the data, while the software
records the time stamp of each modality for follow-up data
alignment and readings, and monitors multiple modalities
during data collection.
The ultrasound armband transducers are placed in one
3D-printed bracket and are worn around the forearm, whose
structure is depicted in Figure 4(c). To ensure that the
bracket's stability and adaptability to different individuals,
eight brackets are connected using an elastic-woven
material, and the transducers are pressed hermetically
against the skin by preloaded springs. The main parameters
of the ultrasound device and ultrasound armband are
listed in Table 1.
The structure of the force-measurement device is shown
in Figure 4(b). Specifically, two force sensors are used to
measure hand force
01 .
^h+ 10 N By the structure of the
measurement device, the hand force is the sum of the measured
values of these two force sensors. The guide rail and
sliding can make sure that the gripping force is exerted on
these two force sensors through the force-bearing link.
Data Processing
The designed multimodal system involves three signal types.
The measured sEMG and ultrasound signals are used to estimate
hand force, and the measured hand force is treated as
the ground truth. The signals from these multiple modalities
should be aligned during the data preprocessing stage.
Table 1. Key parameters of
the ultrasound armband.
Parameter Value
# 4
Repetition
frequency
Damp
Gain
Center
frequency
Vertical
linearity
# 3 %
25 10 kHz
+
400 X
80 dB
5 MHz
Horizontal
linearity
Parameter
Bandwidth
Pulsewidth
Value
+
55 ns
Voltage of excitation -250 V
Measurement depth 50 mm
Velocity of
ultrasound
1,400 m/s
Size of transducer 520mm#z
05 . 15MHz
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
13

IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022