IEEE Robotics & Automation Magazine - March 2022 - 55

to match the robot; 3) then, given a retargeting multiobjective,
we can solve an optimization problem that finds the weights
that reproduce the users' poses. This involves solving a bilevel
optimization problem that can be done, for example, using
sensitivity analysis [15]. A preliminary study showed that
there is variability to the set of weights optimized for every
user. It remains to be seen what the exact distribution of
weights is, across different users, and whether a single average
set will be effective.
Task Autocorrection
Despite many advances, demonstrations show that state-ofthe
art systems, while impressive, are far from exhibiting
human dexterity, and it is clear that a totally immersive experience
will not be attainable in the foreseeable future. One of
the issues that will certainly linger is the difference between
human and robot morphology. Even with personalized retargeting,
this difference will inhibit the sense of body ownership.
However, robots can autonomously perform many
simple tasks, such as pick and place, that human operators
struggle with. Therefore, the question is how to bridge this
gap. One approach is known as assistive teleoperation. As
described in [16], it is the process of arbitrating, i.e., blending,
the operator's motion with a learned optimal policy for a specific
task. Thus, if the task is known in advance, the system
can retarget the motions by making subtle tweaks to the user's
inputs so that his or her maneuvers successfully accomplish
the objective and the modifications are not too intrusive. We
build on this concept and propose a new concept, which we
term task autocorrection, which is similar but geared more
toward immersion. Task autocorrection refers to the process
of making small modifications to the movements of the avatar
such that they match the intention of the operator.
The autocorrection framework consists of several components.
In addition to the teleoperation setup, there is a retargeter
that maps input motions to the
avatar, an intent predictor that outputs
the user's intent, and the task corrector
that adapts the retargeter based on the
predicted intent. The retargeter and
predictor provide continuous output as
the user controls the robot. The task
corrector computes the optimal robot
motion for the predicted task and
blends the user's motion with the optimal
one, based on the confidence level
of the prediction.
As a proof of concept, we devised a
y
x
z
simple deep learning-based autocorrection
prototype for the holographic teleoperation
setup described previously.
In this system, the operator uses a
HoloLens 2 to manipulate a holographic
robotic arm in a scene consisting of
the robot, two desks, and several balls
[Figure 10(a)]. The tasks that were
(a)
considered were picking up and placing the balls and sliding a
grasped ball between two walls. The input data for training the
intent predictor was the hand palm trajectory, the trajectories
of the 26 hand joints, the gaze (origin and direction) trajectory,
and the stacked scene data. These were labeled by the action
itself (e.g., grab and release) and the object it was applied to
(e.g., a blue ball).
The input is transformed into four equal-sized embedded
vectors by forward propagation along two stacked long shortterm
memory layers followed by one ReLU layer in each
branch. The embedding vectors are fused by learnable weighted
averaging and fed into a final ReLU network before computing
the softmax output. The output is a vector, which
represents the probability for each action. For the prototype,
the actions were " pick, " " place, " " slide between walls, " and
" none. " In case a pick or place action was predicted, the target
indicated which object the action was applied to. The training
data set was generated by recording users performing these
tasks, e.g., picking up a ball and randomly placing it on a desk
and picking up a ball and sliding it between two walls. Around
5 h of training data were gathered. The trained model achieved
89.48% accuracy for action prediction and 85.02% for target
prediction on the test set. Next, for each action, an optimal trajectory
was defined. For example, to place a ball on a desk, a
smooth motion that does not penetrate the desk must be computed.
Sliding a ball between the two walls must be done in a
perfect linear motion. The optimal trajectory then arbitrates
the operator's motion based on the confidence level and an
aggressiveness factor that is experimentally determined.
Autocorrection User Study
To evaluate the performance of the prototype, we held a small
user study (n = 7), where participants were invited to interact
with our system, with or without the assistance of autocorrection.
Similar studies have compared the usability of several
Autocorrection
No Autocorrection
z
x
(b)
Figure 10. Autocorrection makes it easier for the user to perform tasks accurately.
Without autocorrection, the user might place the ball through the table. With
autocorrection, the system identifies the task and assists with the optimal motion.
(a) The scene. (b) The task performed with autocorrection on and off.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
55
y
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
