IEEE Robotics & Automation Magazine - March 2022 - 54

These methods commonly formulate an optimization problem
that minimizes combinations and variations of objectives
similar to the ones in the preceding. For example, Rakita et al.
[14] used a weighted sum of an IK objective and other
smoothness terms, while the authors of [13] minimized a
weighted sum of differences among operator finger and robot
finger distances, both with heuristically determined weights.
We conducted experiments to evaluate these retargeting
(a)
methods and found that the technique of choice, and the ideal
parameters for it, change drastically among users, to the point
where the choice of one user is absolutely unworkable by
another. Retargeting does not result in what they perceive as
the motion they aimed to perform, and thus it does not lead
to the desired degree of body ownership. One of the main reasons
is that simplistic retargeting approaches do not faithfully
convey the operator's intent. The setup to test this is presented
in Figure 9, where we use a system consisting of an Allegro
hand mounted on a UR5. The operator could see a holographic
digital twin of the system overlayed over his or her
hand, which provided some visual feedback, as well as the real
robot in the background. The users were asked to manipulate
the hand and perform a pick-and-place task with different
objectives and parameters. Following the experiment, they
were asked to describe their experience, including sources of
frustration in operating the system. In the ensuing discussions,
it became evident that various users have widely differing
preferences for the retargeting model. The hypothesis is
that personalizing the model to the user will have him or her
perform tasks faster and more accurately and reduce mental
strain (evaluated using, e.g., the NASA Task Load Index).
The HoloLens provides a new and exciting methodology
(b)
Figure 8. Retargeting hand poses can be accomplished with
many different strategies. The simpler strategies involve directly
mapping the operator's joint angle to the robot and using the
operator fingertips as IK targets. (a) Angle mapping. (b) IK-based
mapping.
for experimenting and optimizing the user experience of
retargeting. This is thanks to the ability to overlay a hologram
of the robot directly over the operator's body, both sharing the
same frame of reference, in contrast to the operator watching
the real robot react to her motions from an entirely different
perspective, which we observed to be quite confusing for
many people. The methodology we propose for user-specific
retargeting is as follows: 1) users are shown different robot
poses as holograms; 2) they then attempt to pose themselves
(a)
(b)
(c)
Figure 9. Immersive teleoperation by embodiment of a system comprising an Allegro hand mounted on a UR5. The system tracks
the motions of the operator shown in the image using a HoloLens. This data is continuously retargeted and submitted to the robot.
(a) The operator moves into a pose that will cause the arm to grab the block on the table. (b) The operator changes pose to lift the
cube. (c) The inside view of the Hololens, showing the robot hologram superimposed over the user's hand, with the real robot in the
background.
54 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022