IEEE Robotics & Automation Magazine - December 2022 - 28

In this work, we complement the advantages of visual
and force sensing and propose a hybrid framework combining
visual servoing, human motion prediction, and
force feedback. The purpose of our proposed control is to
solve the issue that there exist delays caused by various situations
that lead to unsynchronized motion and large
interaction forces in the scenario of a human-robot
cotransporting task. The application of our proposed control
can reduce the delay time, achieve human-robot synchronization,
and reduce the interaction force between
human and robot.
Dynamics of the Human-Robot
Cotransporting System
In the task under study, the human hand and robotic gripper
hold the object jointly. We consider the dynamic
model of an n-link robot and the object in the joint space
as follows:
MC G
ii
where ,i i ,o and Rn
() (, )(),
i !p
++ =po
o
i !
ii ii xxr
acceleration vectors, respectively. M () Rnn
ii !ioo
#
force. x ! Rn
input torque and the external torque.
(1)
are the robot joint angle, velocity, and
is the inertia
matrix, and it is symmetric and uniformly positive definite.
C (, ) Rn
G () Rn
is the vector of centripetal and Coriolis forces.
and x ! denote the vectors of the control
i ! denotes the vector of the bounded gravitational
r Rn
Hybrid Visual-Haptic Framework
Visual Sensing of Human Motion
In the " pick and place " task, the real-time angle of a human
elbow joint can be obtained by the robot's visual sensing.
A Kinect 2 depth camera is utilized in the task, and project
" BodyBasics-D2D " of the Kinect 2 depth camera is utilized in
obtaining position information of the human body. Pixel
points of human joints can be obtained in real time, and
depth maps with the color camera picture are matched to get
3D positions in camera coordinate systems. Joint points of the
human body are first drawn in real time and then connected
by line segments through Kinect's internal function to obtain
a real-time human skeleton image. As seen in Figure 2, the
locations of the right shoulder (A), elbow (B), and wrist (C) in
the task space can be identified by Kinect 2. Right shoulder
location A and right elbow location B can generate one space
vector
AB , while right elbow location B and right wrist location
C can form another space vector BC . These vectors
define the elbow joint angle, which is confined on the xyplane,
and we define it as
ih .
Human Motion Prediction
Human behavior is continuous when executing a transportation
task, and this article designs a linear predictor to estimate
the human elbow angle in the future, so that the robot can be
controlled in advance to reduce the delay between the robot
and the human. In this part, we define ()thi
as the actual
angle of the human elbow at time t, () as the fitting
as the predicwhere
i is
ih tiTt
value
of the human elbow angle before time iT using the
human motion predictor at time t, and ()thoit
tion angle of the human elbow at time t for the future. LR is
utilized for fitting the historical data of (),tiT
the serial number of past measured values. xi
ih
Shoulder
Right
A
Elbow Joint Angle θh
Elbow Right
B
C
Wrist Right
Z
i
-
is the past
time series, and we always set xi .= a and b are unknown
time-varying parameters of the linear predictor. Square loss
is a commonly used index to evaluate the fitting performance,
and we use the error of the past data fitting value
() () ()
u -= -- -
t
iii
tiTt iT
m
Et iT
=0
2E t t
a
Y
X
ab
2t cmi
ab
(, )
(, )
t t
ab hh
i
2E t t(, )
2b
t (( (( ))
i =0
20
20
=- -=-
-- -=
=- -- -=
/
t
ax hi
i==
ii
i
m
m
2
00i
m
mb
i tm iT ax ))hi (2)
t
,
with respect
t
is obtained by
iho ()t =
t
t
//
/
tm iT bx
t tiT 2
t
to at and bt to be zero.
The predicted angle value
t
ax ()
Figure 2. The human elbow joint angle ih obtained by
visual sensing.
28 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
a and b, which are expressed as at and :bt
(( )( )) ,
(( () )) ,
hhhtiT to find the estimators of
where we consider the partial derivatives of E(, )abt
t
iho ()t
t
p tb+ ;; to ensure () () minimum after
xp denotes the adaptation weight determined by p, which
ii
hho+- t
getting , ,abt
t where p denotes prediction serial numbers, and
tpTt
updates continuously by the gradient descent method:
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
