IEEE Robotics & Automation Magazine - March 2022 - 47

still has a strict requirement that both the user and the robot
are present and available every time a mission must be
defined. The inability to change saved missions also leads to a
situation where missions should be repeatedly executed for
the process of defining them to be worth the effort. Otherwise,
the robot could be used in teleoperation mode with the
same level of efficiency.
Mission Planning
We propose to use MR as a tool for defining inspection missions
in context, without the requirement of first teleoperating
the robot through the desired trajectory. This concept is illustrated
in Figure 1(a). It is made possible by using a common
coordinate system that persists in space and through time to
share spatial information about the mission among devices.
In particular, we leverage the cloud-based localization service
Azure Spatial Anchors (ASA) to create such a reference coordinate
frame (and enable localization to it by other devices)
and spatial data persistence. In this system, a human user with
a HoloLens defines an inspection mission by placing holographic
markers to indicate desired poses for the robot. At a
later time, an autonomous robot can localize to the spatial
anchors that were placed during mission creation; obtain the
set of waypoints and inspection poses, which are defined relative
to these anchors; and autonomously execute the assignment.
This approach has the advantage that the robot, whose
availability is likely a bottleneck in current work environments,
is required only for executing the mission, not for
defining it. Furthermore, these missions can also be edited in
MR, without the need to recreate an entirely new trajectory
every time the tasks need to be adapted.
In practice, the workflow for this system proceeds as follows.
The HoloLens user moves around her environment,
manipulating holograms representing inspection waypoints.
These waypoints can be created, deleted, and connected in an
arbitrary graph structure, including branching. Since localization
to spatial anchors depends on observing the same part of
the environment where the anchor was created, anchors are
automatically fashioned as the user moves away from existing
ones, ensuring that waypoints are always defined with respect
to an anchor to which they are in close proximity. We use a
radius of 2.5 m from previous anchors as the threshold for
creating a new one, which was empirically determined.
Anchor localization accuracy and recall degrade beyond
4-5 m, but an appropriate value here would depend on the
structure and appearance of the environment. Once the user
is satisfied with the waypoints, the mission can be saved by
serializing it and storing these parameters in a cloud-based
database. The user experience during mission planning is
shown in Figure 2(a), where the waypoints are represented by
spheres, with camera frustums for the orientation of the
inspection pose and coordinate axes representing spatial
anchor reference frames.
When the robot is ready to execute the mission, it obtains
the serialized parameters from the database, localizes to the
spatial anchors, and then proceeds through the waypoints. If
the HoloLens user is colocalized to the anchors and running
the app, she can monitor the progress of the mission in MR,
with an articulated model of the robot overlaid on its pose in
the real world [see Figure 2(b)]. Branching decisions at interconnected
waypoints can be made through application programming
interface (API) calls based on the results of the
inspection and by the user selecting the desired branch if the
mission is being executed in this interactive mode with the
HoloLens. The structure of a mission is conceptualized in
Figure 1(b).
ASA
ASA is an MR cloud service designed for localization. The
fundamental concept of the service is that a small visual map,
Inspection Poses
HoloLens 2
Two Interaction
Multiple Spatial
Anchors
List of
Nodes and Edges
Branches With Decision Strategy
(Human, Rule Based, and API Requests)
(a)
(b)
Figure 1. The (a) mission planning workflow and (b) mission components. A human user with a HoloLens device moves through the
environment to be inspected and places holograms that represent waypoints defining a trajectory. Inspection poses can be defined to
trigger the robot to capture data at regions of interest. The underlying structure of a mission is a list of nodes and edges representing
a graph of waypoints, with branching decisions in the graph handled by one of several strategies. As the user moves through the
space and places waypoints, Azure Spatial Anchors are automatically placed to cover the trajectory so that each robot pose can be
defined with respect to a nearby reference coordinate system. This mission structure is then serialized in a JavaScript Object Notation
format so that it can be retrieved from a database by the robot and executed autonomously after localizing to the spatial anchors in
the mission. API: application pro gramming interface.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
47

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022