IEEE Robotics & Automation Magazine - March 2022 - 81

sensors were used to interact with the simulation. The first was
the optical hand tracking device: Leap Motion hand controller
(LMHC). Equipped with two infrared cameras at 120 Hz and
with a 135c field of view (FoV), the LMHC enabled us to interface
between a user's hand movements and the physics engine,
PhysX, in Unity3D. For visual feedback, high-resolution displays
were employed to limit distance overestimation and
degraded longitudinal control, a known issue in VEs [1], [2].
Consequently, a VR head-mounted display (VRHMD), the
HTC Vive Pro, was used, with a 2,880 × 1,600-pixel display
and a 110-FoVc
at 90 Hz. The photosensors on the VRHMD
also represented our second sensor, facilitating head rotations
in the VE.
Robot Operating System was used to import physics
models of robots and objects (in the Unified Robot Description
Format). For all experiments, the LMHC was fixed on
the front of the VRHMD. The physics simulation time step
was set at 1,000 Hz to ensure robust and stable performance
with realistic forces and frictions. Finally, to ensure optimal
hand tracking, lightning conditions were consistent, and the
operational space was limited to about 100 cm as the upper
maximum reaching bounds from the chest of users. In addition,
a low-pass filter with a cutoff frequency of 10 Hz was
applied to the LMHC to reduce noise during retargeting,
ensuring continuity.
Hand Control and Input Interface
As shown in Figure 3(a), a user's hand movements were
mapped to an anthropomorphic Shadow Robot hand in the
simulation. The palm of the simulated hand had 6 DoF and
could move freely around the VE on all axes for translation
and rotation. To telecontrol the virtual hand, the Cartesian
hand movements and joint positions from a participant were
mapped to that in the VE. The retargeting approach was similar
to the work in [14], where the joint positions of a user's
fingers were obtained by calculating the angle i between a
joint bi 1and
its parent joint b .i The resultant angle i from
a user's finger was then incorporated in a joint proportional-
derivative controller to achieve the desired retargeting joint
motions from the user's hand to the simulated robotic one.
These are formulated as
i eo1
$
xi ii
=
iP dc Di
1
=- -() ,$$o
KK
ii
ii
arccos
bb
bb
-
,
-
ii ii
(7)
where i represents the desired angle for the virtual hand to
match. Furthermore, x are the torques applied to each joint i,
and di
i and ci
i are the desired angles between the human
is the
hand and the LMHC and the current angle of the virtual
hand joint in the simulation, respectively. Finally, iio
measured velocity of the virtual hand for computing the
damping torque. A velocity control signal was applied to the
palm of the robotic hand, based on the real hand's position
and orientation to control the 6 DoF of the robotic hand in
the VE. The collision between the robotic hand and the
objects in the simulation was realized via the PhysX 4.0
engine in Unity3D. We also retained the original joint limits
and colliders of the virtual hand, as specified in the Unified
Robot Description Format file of the Shadow hand to ensure
optimal and realistic actuation.
Task Design and Spatial Assessment
In each task, participants were asked to move an object to a
target location. Contrary to the sphere or the index finger of a
participant in most related work [1], [12], [13], the use of a
cube enabled us to assess rotational variations. Due to its
identifiable orientation and as one of the most basic 3D
shapes, the cube presented a suitable choice to assess translational
and rotational tasks. Regarding rotation, we instructed
participants to match the sides of the cube with that of the
cube target, as parallel as possible. While using a cube introduces,
in essence, four " correct " rotations and limits to some
extent the range of rotations one can investigate (e.g., to a
maximum of 45º), it still represents the dominant and most
widely used 3D shape in current work [14]. Our approach is
influenced by [10] and [11], which included rotational tasks
but were limited to 2D movements. The targets were
arranged in spherical coordinates with the object at the center.
Figures 1 and 2 illustrate movements in full 3D and in 2D
space, respectively.
The object manipulated by users was presented as a solid
blue cube and the target location as a transparent cubic volume.
When intersected and translational and/or rotational
requirements met, depending on the task type and difficulty,
the transparent target would turn green, indicating success
and progressing to the next, as shown in Figure 3(b) and (c).
For translational tasks, a 50% overlap with the target was considered
a success, as with Fitts's original experiment. For rotational
tasks, the target rotation, ,a needed to be matched
within a certain rotation tolerance,
~ , on all axes. In Figure 4,
we describe the equations that needed to be satisfied for a task
to be classified as a success or an error.
Pointing Versus Manipulation
In addition to varying all possible spatial variables, we investigated
pointing and manipulation tasks, which are the most
common types of interactions in VEs [1] and teleoperation
[14], [16]. Pointing tasks were investigated, as they closely follow
Fitts's original experiment, represent the dominant types
of interaction in 3D user interfaces, and resemble peg insertion
tasks in teleoperation [16]. To cover the critical limitation
in the current literature for using Fitts's law for robotics, we
studied manipulation tasks, e.g., using grasping, as they are
the dominant interactions in teleoperation [2]. For pointing
tasks, we used the index finger pad of the virtual hand to
" attach " the cube to a participant's hand, enabling users to
move their hand and the cube to the target location. The
object would attach or collide during the intersection with the
index finger, and it would match the position of the pad but
retain its original orientation. On the other hand, for the
manipulation tasks, realistic physics and friction forces were
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
81
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
