IEEE Robotics & Automation Magazine - March 2022 - 49

WebSocket. An additional ROS package of utilities (https://
github.com/EricVoll/spot-mr-core) helps to interface the Spot
with the ASA service. The robot's behavior is orchestrated by
the ASA ROS commander node, which obtains the mission
definition from either ROS or the cloud database. Once the
robot localizes to the spatial anchors defined in the mission
using the ASA service, this node sends the waypoints as trajectory
commands through the wrapper to the Spot SDK and
monitors the robot's progress through the mission. An important
note is that this mission planning system relies on the
robot's underlying navigation capabilities to actually plan and
execute paths to the waypoints that are sent as goals.
Finally, the cloud components consist of the ASA service
(see the " ASA " section) and a CosmosDB database to store
serialized mission parameters. The waypoint poses, spatial
anchor IDs, and other mission information are serialized into
JavaScript Object Notation (JSON) format and stored in the
database with a unique ID as a key. When operating in the
temporally decoupled mode, the ASA ROS Commander
node can look up a particular mission in the database, download
the JSON, and then execute the mission once it is localized
to the mission's anchors.
Outlook
This system represents a milestone in spatial computing and
interaction. By sharing common reference frames, the MR
device can provide the robot with actionable spatial information
that the robot can understand in its own spatial context,
leading to an improvement in efficiency in common commercial
and industrial robotics tasks. One limitation of the current
spatial anchors service is that the anchor maps are not automatically
connected in the cloud to form large-scale, continuous
digital twins even if they are located in the same area. This
use of discrete spatial anchors limits scalability for this application,
but future work on using continuous digital twins for the
localization of multiple heterogeneous devices will enable this
type of shared spatial understanding on a large scale.
Colocalization and Interaction
The colocalization of devices requires that they are able to localize
themselves to a common reference coordinate system.
Through their individual poses with respect to this common
coordinate frame, the relative transformation between localized
devices can be computed and used to enable new behaviors and
collaboration between devices. In the scenario described in the
" Sharing Spatial Information " section, the robot and MR device
do not directly interact and are not necessarily colocalized at
the same time, but they share spatial information that is
anchored to a common spatial reference frame. Here, we consider
a scenario in which multiple devices simultaneously colocalize
to a shared coordinate system, thereby enabling
temporally synchronized behaviors and interaction.
2
Intel NUC, Ubuntu 18.04
3
ROS Bridge (C++)
ROS
ASA ROS Commander (Python)
ROS
Spot ROS Wrapper (Python)
1
gRPC API, Ethernet
Boston Dynamics Spot
5
Human
Holograms, Robot Model,
and MR Interaction
Local Network
WebSocket
4
Unity
ROS# UWP (C#)
ROS#
Mission Planning and Execution (C#)
Figure 3. The components of the mission planning framework. (1) The robot communicates through its SDK to an ROS wrapper
and our mission planning orchestration node, which (2) run on a companion computer. This computer communicates with the
HoloLens across (3) the local network and through serialized missions stored in an Azure database. The HoloLens interfaces with
the ROS communication framework using (4) an ROS# Unity plugin to the mission planning application, which ultimately provides
an interactive MR experience for (5) the user. NUC: Next Unit of Computing; gRPC: Google Remote Procedure Call; UWP: Universal
Windows Platform.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
49

https://github.com/EricVoll/spot-mr-core https://github.com/EricVoll/spot-mr-core

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022