IEEE Robotics & Automation Magazine - March 2022 - 48

created through structure from motion and representing a
specific location in the world, is stored in the cloud with a
unique ID. The set of features and descriptors constituting
this map then define a coordinate system that is fixed to the
world; if those features can be detected later and their 3D
positions can be recovered, the coordinate system can also be
recaptured. Devices wishing to localize to a particular anchor,
known by its unique ID, then create their own structurefrom-motion
map, which they submit to the cloud service as
a query. If correspondences for the feature points in the query
can be found in the reference map in the cloud, the service
returns a relative pose.
This service is cross platform and relies on a device's
onboard visual odometry to build these visual maps to create
and query an anchor. On HoloLens devices, the onboard
head-tracking processes handle this. For mobile devices,
ARKit and ARCore provide the image features and poses for
iOS and Android, respectively. On robotic systems, a special
research version of the software development kit (SDK) was
released with Ubuntu Linux support as well as a Robot Operating
System (ROS) wrapper (https://github.com/microsoft/
azure_spatial_anchors_ros). Unlike other devices, where the
visual pose estimation happens in the context of a platformspecific
odometry system (e.g., an ARCore or ARKit session),
pose estimation solutions on mobile robots can be quite
diverse. The Linux version of the SDK thus accepts undistorted
images and poses and computes its own features. The pose
of the camera can be directly estimated through visual odometry
on the robot and by attaching the camera with a calibrated
transformation to a mobile base that estimates its pose
through some other means (e.g., with lidar). While the success
of localization queries to ASA depends heavily on the
particular cameras, trajectories, and environments in question,
in practice, the performance is in the range of centimeter-level
accuracy. This is documented in Figure 2(b), where
the robot's estimate of its pose with respect to the anchor
2×
closely matches the actual robot pose when it is transformed
into the HoloLens's field of view, as seen in the close alignment
of the robot and its holographic model.
System Overview
The system is organized in a modular and distributed fashion,
with components deployed on the HoloLens, on the robot,
and in the cloud. It is possible to run the HoloLens and robot
at the same time, where the mission parameters and some
visualizations are sent between the devices with ROS, and to
define and run the mission at different times, in which case
the mission definition is serialized and stored in a databse in
the Azure cloud. An overview of these components and the
flow of data among them appears in Figure 3.
The HoloLens runs an application that is built using the
Unity 3D engine (https://unity.com/). Some prebuilt components
from the MR Toolkit (https://github.com/microsoft/
MixedRealityToolkit-Unity) handle the user's interaction with
the app's holograms. The ROS# (https://github.com/siemens/
ros-sharp) package acts as a bridge between the Unity environment
and ROS communication framework. HoloLens
provides support for creating and querying ASA through an
SDK, and this functionality is connected to Unity through an
XR plugin layer. These components work together to enable
the user to create and move holographic markers in the environment
and store the poses of these waypoints with respect
to spatial anchor coordinate systems.
On our robot, a Boston Dynamics Spot, we connect a
companion computer (an Intel NUC running Ubuntu 18.04)
through the robot's ethernet port to run our user code. This
code includes a wrapper for Spot's Google Remote Procedure
Call-based SDK (https://github.com/boston-dynamics/spot
-sdk) that connects its data streams to ROS topics (https://
github.com/clearpathrobotics/spot_ros) as well as a corresponding
bridge node that receives messages from the ROS#
component in the HoloLens Unity app that are sent over a
2×
(a)
(b)
Figure 2. Mission planning and execution. (a) The view through the HoloLens as the user moves holographic markers for the
trajectory waypoints. Each waypoint is represented by a sphere for the desired position and a camera frustum to define the
orientation the robot should take for inspection. The axes to the right of the trajectory mark the origin of an automatically created
spatial anchor coordinate system. (b) The user's view as the robot executes the mission, with a robot model overlaid on the physical
one and a sample data capture at an inspection pose. This user view during mission execution is only for illustration, as the robot can
autonomously execute the mission without human intervention.
48 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022

https://www.unity.com/ https://github.com/microsoft/MixedRealityToolkit-Unity https://github.com/microsoft/MixedRealityToolkit-Unity https://github.com/siemens/ros-sharp https://github.com/siemens/ros-sharp https://github.com/microsoft/azure_spatial_anchors_ros https://github.com/microsoft/azure_spatial_anchors_ros https://www.github.com/boston-dynamics/spot-sdk https://www.github.com/boston-dynamics/spot-sdk https://github.com/clearpathrobotics/spot_ros https://github.com/clearpathrobotics/spot_ros

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022