IEEE Robotics & Automation Magazine - March 2022 - 61

collected data from the subscriber and renders corresponding
visual elements at 20 Hz. This callback frequency was
chosen to suit the limited processing capabilities of AR headsets,
while being frequent enough to maintain a smooth
motion of the visual elements-only slightly slower than the
24 Hz frame rate for movies.
Tool Plugins are components of ARviz that utilize input
modalities of the AR device such as gaze tracking and speech
recognition to provide a multimodal communication interface
with ROS. Tool Plugins can be used for communicating with a
robot's navigation stack to move the robot around the environment
or for issuing command to a robotic manipulator to start
a handover task (more details are provided in the " Mobile
Robot Navigation " and " Visualizing Robotic Arm's Goal Pose
During Object's Handover " section). Similar to Display Plugins,
users can implement their own plugins to tailor to their needs
for communicating with ROS and interacting with the robot.
The implementation of this plugin can be varied depending
on the communication channel. The Tool Plugin is developed
as an extension of the input handler provided by
Microsoft's Mixed Reality Toolkit (https://docs.microsoft
.com/en-gb/windows/mixed-reality/mrtk-unity/?view=
mrtkunity-2021-05), a Unity package for mixed reality application
development, which provides APIs to input system and
other building blocks for spatial interactions. User inputs are
collected by Tool Plugins, which then send the information to
ROS using ROS#, through a component called a publisher
(more details in the " Communication with ROS " section).
Hand Menu
A menu that can be summoned at the user's fingertips, literally,
is developed to provide quick and easy control of all of the
available plugins implemented in ARviz. By raising their left
hand as shown in Figure 3, the user can make the hand menu
appear and access configurations of the Display Plugins and
Tool Plugins. Using this menu, users can toggle the visibility of
Display Plugins, enable/disable Tool Plugins or access/modify
each plugin's specific properties (as shown later in Figure 6).
Additional menu entries can be added, and their behaviors
customized for user-defined Display Plugins and Tool Plugins.
Hologram Localization
For AR applications, we need to consider two coordinate
systems: the virtual world coordinate system (VWCS),
where virtual objects/holograms are rendered, and the real
world coordinate system (RWCS), where the physical entities
(e.g., the robot) exist. Typically, when an AR application
is started up, it arbitrarily initializes the base frame of
the VWCS to be where the AR device is physically located
in the real world. On the other hand, in a ROS application,
the RWCS is managed by the tf transformation ROS package,
and the base frame of the RWCS can be defined by the
user to be anywhere in the physical world. In order for the
Display Plugins and Tool Plugins in ARviz to render virtual
display elements at appropriate locations in the real world,
two pieces of information are needed: 1) the pose (position
and orientation) of the VWCS with respect to the RWCS,
i.e., the transformation between the VWCS and the RWCS;
and 2) where the display element should be located with
respect to the RWCS. These two pieces of information are
provided by Vuforia Engine and Unity tf Listener, as
described later.
Aligning RWCS and VWCS using Vuforia
To define the transformation between RWCS and VWCS, we
define an anchor point that tell us where the VWCS base
frame should be, within the RWCS. Vuforia Engine allows the
specification of this anchor through a physical QR code
marker placed in the real world. To create this anchor, a physical
QR code marker is placed at fixed, known location within
the RWCS. In ARviz, the Vuforia Engine is deployed to detect
this QR code marker. Once the AR devices detects the QR
marker, it computes the transformation between the AR
device and the physical QR marker. Since the position of the
AR device with respect to the VWCS is always tracked by the
AR application, the transformation between the VWCS and
RWCS can be computed. Once this transformation is computed,
ARviz then repositions the VWCS base frame to coincide
with the RWCS base frame, aligning the virtual world
with the real world. The alignment process is illustrated in
Figure 4.
(a)
(b)
(c)
Figure 4. ARviz's virtual world alignment. The virtual objects would be initially rendered at random location, misaligning with the real
world (a). With the use of the QR code marker in the real world and Vuforia Engine in ARviz (b), the virtual world is aligned with the
real world, allowing the correct overlay of the virtual objects onto the real world (c).
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
61

https://docs.microsoft.com/en-gb/windows/mixed-reality/mrtk-unity/?view= mrtkunity-2021-05 https://docs.microsoft.com/en-gb/windows/mixed-reality/mrtk-unity/?view= mrtkunity-2021-05 https://docs.microsoft.com/en-gb/windows/mixed-reality/mrtk-unity/?view= mrtkunity-2021-05

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022