IEEE Robotics & Automation Magazine - March 2022 - 46

Digital Representation
Since the term was first coined almost 20 years ago [1], the
rise of virtual, augmented, and MR technologies has highlighted
the importance and applications of spatial computing
as a novel paradigm for interacting with spatial information
through our devices. Spatial computing refers to the digitization
and modeling of the device's environment and the
objects within it such that the device has spatial context.
What distinguishes MR from virtual reality is the ability to
simultaneously observe the physical and digital worlds, with
digital content aligned to the real spatial environment, and
the ability to interact with physical and digital objects. While
augmented reality (AR) devices offer some of these capabilities
through touchscreens, MR is characterized by more
immersive visualization and interaction through headmounted
displays.
In the world of mobile robotics, spatial computing is a
requirement for most operations. To navigate an environment,
avoid obstacles, and perform useful functions, mobile
robots need to build and use a digital representation of their
understanding of the area. Often, this takes the form of a map,
where the robot maintains an estimate of where it is in space,
as well as the structure of the environment. More broadly, this
representation can be a digital twin of the environment,
where any data about the world that has spatial meaning can
be embedded in a digital framework intended to capture
information about the space and the digital devices in it as
accurately and completely as possible. The trend in devices
taking advantage of spatial computing is often called the third
wave of personal computing, after the first two waves, where
desktops provided access to home computing before mobile
devices initially made this power ubiquitous and portable,
although the devices lacked awareness of the space they were
in. Mobile robots and MR devices thus have significant synergies
since they require many of the same spatial computing
capabilities, regardless of whether they are designed to be
used by a human or behave autonomously.
This alignment of spatial understanding between humanoriented
MR devices and robots provides an opportunity to
unlock new modes of interaction between people and
machines. In particular, by sharing these spatial representations
and digital twins among human and robotic devices, we
can enable all the agents involved to leverage that information
for greater capabilities. This enables them to do more and
provides a common understanding so that humans and
robots can work together more efficiently through natural
and intuitive interaction and collaboration. Additionally, by
leveraging the egocentric sensing and immersive visualization
of MR devices, we can provide embodied teleoperation experiences
for remote devices. In this scenario, the user projects
his or her actions to a remote robot while using the robot's
spatial understanding to provide immersive feedback about
the task at hand. In this article, we aim to show our efforts in
this domain, with three works that illustrate how MR can
enable more flexible collaboration and even new types of
interactions between humans and robots.
46 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
Sharing Spatial Information
We first consider a scenario in which devices share spatial
information in a temporally decoupled fashion. Here, we
demonstrate that spatial data can be persisted through time
by defining them with respect to world-locked reference
coordinate systems. The general workflow begins with one
device defining a reference coordinate system by building a
visual map of the environment. Spatial information is then
anchored to this reference frame, which is fixed with respect
to the space, enabling it to persist in that location. Another
device, or the first device at a later time, can then relocalize to
this map, recover the reference coordinate system, and access
the stored information in the same place in the world where it
was defined.
In this section, we show how this type of workflow can be
utilized to enable the planning of robotic inspection missions.
This work is motivated by the need for automated inspection
in many commercial and industrial settings, where mobile
robots have the navigation capabilities to actually execute the
missions but where planning the path that a robot should take
is still a cumbersome process. In these large, dynamic environments
and in unconstrained spaces, such as disaster zones,
it is often not possible to augment the environment with fiducial
markers to facilitate the sharing such spatial information
from the planner to the robot. We therefore propose to use
shared, world-locked coordinate systems through MR as a
way to provide a common spatial reference.
Existing commercial solutions for mission planning either
use a computer interface that is decoupled from the environment
or require the user to drive the robot through the trajectory
first, in a teach-and-repeat fashion. Planning a robot
trajectory within a high-fidelity 3D model or mesh of the
environment using a computer is certainly possible and currently
deployed in commercial services [e.g., Rocos (https://
www.rocos.io/content/robotics-fleet-management)]. However,
this approach requires the model to contain a high level of
detail about the inspection targets to accurately indicate the
area the robot should observe without actually being there.
This type of approach has been used for ground robot [2]
and drone mission planning in mixed [3] and virtual reality
[4] by using 3D terrain models, which are far more available
than high-resolution digital twins of inspection sites for
ground robots. Additionally, AR [5] and MR [6] have been
used with a virtual robot arm to facilitate trajectory programming,
but connecting these simulated arms to a real
robot has used fiducial markers rather than colocalization
with any spatial context.
Unlike these prior works, utilizing the spatial context of
the environment provides a more natural way to define a
robot trajectory. However, while solutions such as Boston
Dynamics' Autowalk feature for Spot enable the user to plan
missions in context while physically present in the space, such
a teach-and-repeat approach requires the user to manually
control the robot through the environment every time he or
she defines a mission, and it is not possible to edit missions
without rerecording them. While this is a powerful feature, it
https://www.rocos.io/content/robotics-fleet-management https://www.rocos.io/content/robotics-fleet-management
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
