IEEE Robotics & Automation Magazine - December 2022 - 27

Several recent works have been aimed at human-robot
cotransporting and have explored human-robot joint
manipulation. Huang et al. [15] propose a method for single-leader-dual-follower
teleoperation based on force to
estimate the position in the presence of dynamic uncertainties.
Humanoids and mobile robots are utilized to
transport the object in collaboration with humans. In [16],
a machine learning approach based on sensory information
is proposed without force sensors, and a statistical
model of human behavior is developed. Some works focus
on the estimation of unknown cotransported object
dynamics. Cehajic et al. [17] proposes a method to identify
an unknown grasp pose of a human, where the leastsquares
method is utilized to achieve an online estimation
of relative kinematics.
Task assignment is also an important research content in
human-robot collaboration tasks. Some research works
achieve role exchanging by granting the control effort of a
human regarding human motion intentions or systematically
deriving role assignment from the geometric and
dynamic task properties. Some strategies on the assignment
of task effort are proposed in [18], in which a controller is
introduced that enables humanoid robots to perform a
cotransporting task with the human. Robots can estimate
the human motion intentions to assist the human, and then
robots take over the leadership of the task to perform it. In
[19], the authors propose a method by which the robot can
learn reactive and proactive behaviors in the joint task by
learning from demonstrations.
Most of the aforementioned control approaches use
force information to generate passive behaviors for the
robot, which inherently brings a burden to the human.
Some works have considered a hybrid visual-haptic
framework, which combines vision and haptic information,
with the state of the task obtained from visual sensing.
Inspired by these works, we have integrated vision
and haptic sensing to perform a cotransporting task [20].
However, the aforementioned issues of delays have not
been considered.
In this article, we propose a hybrid visual-haptic
framework for a human-robot cotransporting task. A
novel method is proposed to predict human motion for
reducing delays to achieve motion synchronization. Visual
sensing is utilized in capturing human motion in real
time, and robot motion is refined based on force sensing.
Adaptive NN-based control is proposed to achieve
precise trajectory tracking. Comparative experimental
results show that our proposed framework is effective in
a cotransporting task, motion synchronization can be
achieved, and the human participants find it easy to
accomplish the task.
Based on these discussions, we highlight our contributions
as follows:
1) A human motion prediction method is proposed to reduce
delays in a cotransporting task, and a composite filter is
designed to obtain human motion data.
2) A hybrid visual-haptic framework is proposed that integrates
impedance control and visual servoing.
The rest of the article is presented as follows. The next section
presents the dynamics of human-robot collaboration systems,
and the section " Hybrid Visual-Haptic Framework "
introduces visual signal sensing, predicting, and filtering.
The section " Control
Framework " proposes a
hybrid visual-haptic control
framework considering
uncertainties in a
system's dynamics and
also analyzes the system
stability. The proposed
control framework is
In this work, we
complement the
evaluated in the section
" Experiment " by comparative
experiments.
The section " Conclusion "
concludes our work.
Problem Formulation
advantages of visual and
force sensing and propose
a hybrid framework
combining visual servoing,
human motion prediction,
and force feedback.
System Description
Figure 1(b) shows a scenario
including two agents whose goal is to transport an object
from a " starting position " to a " storage position. " Because
of the length and weight of the object, this task cannot be
accomplished individually by a single agent. Inspired by
human-human cotransporting, we introduce a robot collaborator
in the task. In this work, the human plays a leader
role by initiating the movement. The robot plays a follower
role to share the load and collaborates with the human. We
see that large delays between the human and robot may lead
to task failure. We assume that the robot is equipped with
visual and haptic sensors, e.g., a camera and a force torque
(FT) sensor.
Definition 1
Human motion is defined as the position of the human hand,
which contacts the transported object. Robot motion is
defined as the position of the robotic gripper, which grasps
the transported object.
If the robot can predict human motion and track it precisely,
motion synchronization can be achieved. Visual servoing
has been extensively studied, but it is not applicable
to the cotransporting task under study because 1) a large
resistance force will be generated if the human and robot
have asynchronized motion, and 2) human motion prediction
based on vision only is always subject to model overfitting.
Force-based control methods are also utilized in
related works, where robots can regulate the interaction
force and generate different behaviors to cooperate with
humans. However, delays in cooperation are inevitable
based on force feedback only, and this leads to large interaction
forces.
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
27
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
