IEEE Robotics & Automation Magazine - March 2022 - 50

When the colocalized devices are all mobile robots,
swarming and collective behaviors could be possible, as could
the parallelization of spatial tasks. On the other hand, if there
is a mix of human-centered devices (e.g., MR headsets and
mobile devices) and robots, colocalization can unlock more
natural human-robot interaction. This section describes a
system that exploits the colocalization of a human wearing a
Microsoft HoloLens 2 and a mobile robot to demonstrate
intuitive hand gestures for robot control. This work is motivated
by a need for semiautonomous behaviors with shared
control as a way to reduce the attentional load for the operator.
Defining high-level tasks for the robot to perform autonomously,
particularly through intuitive interfaces, enables the
user to focus on other objectives and control multiple devices
simultaneously. This is desirable in search-and-rescue environments
[7] and will be important in the increasingly robotfilled
workplace of the future.
Colocalizing the Human and Robot
We consider two approaches for colocalizing devices to a common
reference frame: sharing a map and utilizing a visual
localization service in the cloud. Prior work in this domain has
explored the use of AR for more efficient human-robot interaction
and visualization of the robot's state and intent. However,
these works have relied on colocalization through the
detection of landmark objects [8] and the use of fiducial markers
mounted on the robot [9]. While these approaches for
colocalization are sufficient to perform AR visualization, they
do not provide further spatial understanding for the AR device
or robot. Here, we propose to share not just a relative pose but
a shared map between the devices, enabling both devices to
take advantage of a common digital twin of the space.
To share a map from the HoloLens with the robot, we execute
an offline procedure to create and then convert a map of
an environment for colocalization. We leverage the onboard
spatial mapping processes of the HoloLens, which constantly
build a sparse visual feature map for tracking the motion of
the device as well as a dense mesh of the environment. The
user observes the environment with the HoloLens depth
camera to build the dense mesh, and our application provides
visual feedback to show areas of the space that have been
mapped [see Figure 4(a)]. The sparse map is aligned to the
dense representation, and it enables the HoloLens to relocalize
to the environment in a future session.
Once the spatial mesh has been captured, we apply several
processing steps to convert it to an occupancy grid representation
that can be used by the robot for lidar-based localization.
We take the mesh as input, which typically consists of several
connected components, and apply Poisson reconstruction to
make it watertight. This watertight mesh is used to initialize a
signed distance function (SDF) representation of the space.
Finally, a horizontal 2D slice from the SDF is extracted at a
user-defined height such that the implicit surfaces in the SDF
define the occupied cells in the occupancy grid representation,
and the voxels in the SDF with positive distance represent free
space cells. An example of such a map being used for lidar
localization is provided in Figure 4(b). The xy origin of the
map is preserved during these conversion steps, so 2D coordinates
given in the HoloLens coordinate frame correspond to
the same xy position in the plane where the robot is navigating.
In the shared map scenario, when the HoloLens and the
robot localize to their respective maps, spatial information can
be translated between the two map representations.
We also demonstrate the use of the ASA cloud localization
service to colocalize the HoloLens and robot to the
same anchor. This service was described in more detail in
the " ASA " section. For this application, we require a single
anchor that both devices can localize to, which then provides
a common reference frame. In our workflow, the
HoloLens observes the environment and creates an anchor
through the service. We synchronize the unique ID of the
created anchor with the robot through ROS, then leverage
the ASA ROS wrapper to query this anchor by using a
stream of camera images and poses from the robot. Once
the robot localizes to the desired anchor, the anchor's reference
frame is added to the ROS tf tree, where it can be used
(a)
(b)
Figure 4. The colocalization of a robot and HoloLens through a shared map. (a) The user's view of a spatial mesh, which was captured
from the HoloLens, overlaid on the real world. (b) This map is converted to a 2D occupancy grid representation, whose coordinate
frame is aligned with that of the mesh, to enable robot localization with lidar.
50 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022