IEEE Robotics & Automation Magazine - March 2022 - 56

non-MR teleoperation methods [17] and provided useful
insights into the effect of the user interface on the helpfulness
and ease of use in manipulation tasks. For other tasks such as
robotic inspection, AR interfaces have demonstrated significant
benefits for users in the speed and performance of robot
teleoperation [18]. The participants were required to grab a
gray ball and place it on a spot marked by an indicator
(shown as a small sphere in the hologram) and then grab a
yellow ball and place it next to another indicator. They were
told to try to place the ball properly; that is, it should not have
been placed in the air or through the surface of the desk. As
evident in Figure 10(b), without autocorrection, due to the
operator's limited capacity to control the robot, the ball is
maneuvered through the desk, but with autocorrection
enabled, it is perfectly placed.
In a separate experiment, participants were asked to pick
the gray ball and twice drag it from side to side along a slot. In
this case, they were told to keep the ball as low as possible
without colliding with the desk and walls. Before starting the
experiment, a warm-up trial was conducted to let the participants
become familiar with the basic operations. For each trial,
the participants were asked to perform either a pick-and-place
or slide task once with or without enabling autocorrection and
without knowing whether the function was enabled or not.
The recorded experiments were evaluated using two
quantitative metrics during simulation: violation distance
and operation time. In addition, the participants were asked
to rate the degree of naturalness, from one being " very
unnatural " to five being " very natural, " during the control for
each trial. A task load index was also rated by participants for
this question, referring to the effort rating of the NASA Task
Load Index, which we rescaled from one, meaning " very
easy, " to five, meaning " very hard. " The hypotheses were that
the autocorrected motions would cut the violation distance,
shorten the operation time, and reduce the task load index
while lowering the naturalness score as well. Analysis shows
that the improvement was statistically significant, with the
downside being the reduced naturalness score. This drawback
could be addressed by using a more refined autocorrection
system.
Outlook
The solutions presented in this section address some of the
challenges with the remote teleoperation of robot arms. Motion
retargeting and task autocorrection make human-robot interaction
feel more natural and help users behave more naturally
through assistance with executing accurate moves. Looking
forward, there are several limitations to the current framework.
The MR device acts as an egocentric sensor platform, capturing
the human's actions, but these works do not yet take advantage
of either the robot's or the human's spatial context.
Extending the proposed system to tasks that require dynamics
will demand physical simulation. A realistic simulation will
further enable gathering data that include different grasps and
more dexterous manipulations tasks. Considering its scalability
to many tasks, another limitation of the proposed approach is
56 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
the necessity to obtain data for each task and define each optimal
trajectory. To this end, a database of tasks and optimal trajectories
must be established, and autocorrection could even be
used to bootstrap it. In addition to improving the generalization
of these methods to more and more arbitrary tasks, future
work to leverage spatial computing will provide immersive and
embodied teleoperation experiences with additional capabilities
to accomplish tasks in real-world environments. Separate
from the addition of new features, an expanded user study with
more participants and deeper experiments would provide valuable
information to guide future work in preparing autocorrection
to be applied to real-world tasks.
Conclusion
This article presented several prototype systems that utilize
robots and MR devices to provide novel solutions to compelling
real-world applications through human-robot interaction.
The two key technologies that enable these solutions are the
spatial computing and egocentric sensing capabilities of MR
devices. All three systems make use of one or both of these to
provide new spatial capabilities as well as intuitive and natural
interaction. However, they are only preliminary explorations
toward these real-world applications, and our initial results
have uncovered more questions than they have answered.
The primary barrier to the practical deployment of the
proposed mission planning framework from the " Sharing
Spatial Information " section is the lack of support for largescale,
continuous maps in ASA (and any other spatial anchor
cloud provider). Ideally, we would like to have our robots and
devices continuously localized as they move through the environment,
not just when they are near a particular anchor.
This capability will require large-scale, continuous maps in
the cloud and a service that provides localization to a venue
(rather than an individual anchor) that is robust to the network
dropouts that may occur in large indoor spaces. The
robots and MR devices should also be able to leverage a priori
spatial data from building information models and CAD data
to visualize and use as-planned information from these
designs and as-built information from physically being on
site. These features are not present in current commercial
localization services, but state-of-the-art research has demonstrated
that these capabilities are ready to be applied to these
real-world scenarios.
On the other hand, there are still many open avenues for
research in human-robot interaction through MR. In particular,
although we have demonstrated intuitive control by giving
a robot commands through natural hand gestures, the
robot's understanding of this interaction is no different than if
these navigation goals were provided by a keyboard and
mouse, and its representation of its environment is still one of
obstacles and free space. While this lack of semantic understanding
is not a barrier for simple tasks, such as navigation to
a goal, further work will be required to endow the robot with
the understanding of human presence and the semantic context
necessary to perform more complex tasks that can be
communicated in a natural way by humans. We have released
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
