IEEE Robotics & Automation Magazine - March 2022 - 8

FROM THE GUEST EDITORS
Extended Reality in Robotics
By Elena De Momi, Mahdi Tavakoli, Jeffrey Delmerico, Antonio Frisoli,
Mark A. Minor, Giovanni Rossini, and Paul Chippendale
E
xtended reality (XR), which
combines the real and virtual
worlds, is greatly enhancing
interaction possibilities between
robots and humans, leading to a
paradigm shift where the two entities
can intuitively cooperate to perform
shared-target tasks. Many XR devices
are essentially performing the same
spatial perception tasks as mobile
robots (e.g., visual simultaneous
localization and mapping), and thus
XR provides an opportunity for robots
and the humans using these de vices
to colocalize through a common un -
derstanding of their space, which also
enables easier
human-robot
interactions.
XR interfaces
can be realized
through augmented
reality
(AR), where an
operator's perception
of the
real world is en -
hanced through
the superimposition
of virtual
spatial intelligence that AR and MR
can provide.
The COVID-19 pandemic has highA
key enabler
of human-robot
collaboration is
the colocalization
and shared spatial
intelligence that AR
and MR can provide.
objects and in formation; virtual reality
(VR), where the operator is immersed
in a 3D virtual world; or mixed reality
(MR), where the user can both see and
interact with digital content that is
superimposed over the real world. A
key enabler of human-robot collaboration
is the colocalization and shared
Digital Object Identifier 10.1109/MRA.2022.3143186
Date of current version: 22 March 2022
lighted an unprecedented need for
immersive platforms that can facilitate
safer working conditions through telerobotic
control. Health-care workers,
for example, can use XR technologies to
enable intuitive collaborations for
remote perception and control, such as
diagnosis, treatment planning, or
patient monitoring. In this context,
research efforts have received a significant
push toward realizing fascinating
promises in several application fields,
with XR being a prime contender to
offer immersive experiences, even from
remote locations. For instance, they can
enable immersive remote inspection in
dangerous environments.
This special issue explores the recent
advances in AR, VR, and MR for
human-robot interaction (HRI) in the
field of robotics.
In the first article, Ranne et al. propose
a novel AR interface for controlling
and visualizing a malleable robot
arm. Because of the adaptive topology
of the variable-stiffness links in these
robots, different reconfigurations of the
malleable link result in unique workspaces,
which can be nonintuitive for the
user to predict based just on the endeffector
pose. The authors leverage AR
to enable the user to manipulate a digital
twin of the robot's end effector to a
desired position and then select a reconfiguration
from several candidates
whose workspaces are visualized
through the head-mounted AR display,
overlaid on the real world. The proposed
8 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
interface allows the user to accurately
align the physical robot to the virtual
one, resulting in the desired reconfiguration
and workspace, without adding
a significant workload for the user.
In the second article, Quesada and
Demiris propose an affordance-aware
proactive planning, whereby an AR user
interface proactively identifies user
affordances, i.e., action options that are
possible in the current context. The tool
combines natural language processing
and planning algorithms to provide the
most relevant and feasible options based
on the user's context as well as to reduce
the time needed for these options to be
generated and presented to the user.
The resulting affordance-aware AR user
interfaces for robot control combine
atomic actions and provide higher-level
options to the user.
In the third article, Groechel et al.
apply the tool for organizing key characteristics
of virtual, augmented, and
mixed reality Technologies in human-
robot interaction (VAM-HRI) framework
to several papers presented in
the fourth VAM-HRI workshop. The
framework introduces a proper taxonomy
and the interaction cube, which captures
characteristics about the design
elements involved (expressivity of the
view and flexibility of control) and the
virtuality they implement (from real to
fully virtual). Metrics for systems evaluation
are presented as well.
In the fourth article, Delmerico et al.
explore several novel applications of
MR for HRI: mission planning for
inspection, gesture-based control, and
immersive teleoperation. Each of these
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
