IEEE Robotics & Automation Magazine - March 2022 - 60

ARviz
Application
Hand Menu
JSON
Over
Tool
Plugins
Display
Plugins
TCP/IP
ROS
Nodes
Figure 2. The basic architecture of ARviz Platform.
interactive interfaces for AR-HRI applications is proposed to
fill an unmet need for the AR-HRI community.
Recognizing this need, we developed ARviz, a versatile
AR visualization platform for ROS-based applications. ARviz
is inspired by Rviz, and allows users to visualize any standard
ROS messages data types in AR. Furthermore, with additional
input modalities, such as speech, gesture, and gaze,
enabled by AR devices, ARviz provides a multimodal interface
for users to interact with any ROS applications. ARviz is
designed to be extendable, allowing users to implement their
own plugins for visualizing customized ROS messages, or for
defining their own interaction methods based on their specific
application needs. By keeping many design similarities
with Rviz, ARviz aims to provide a sense of familiarity for the
large existing user-base of Rviz, as shown in Figure 1. In this
article, we describe ARviz's platform architecture, present the
collection of Display and Tool Plugins included in our current
implementation of ARviz, demonstrate the benefits of
ARviz through a number of use cases featuring different
applications of ARviz, and discuss potential further extensions
and uses cases for ARviz as well as some limitations.
Platform Architecture
ARviz is designed to be a stand-alone AR application that can
connect to any ROS-based application to provide visualization
and interaction capabilities to the user, requiring
minimal-to-no-modifications to the existing ROS application.
Figure 2 depicts the overall platform architecture for ARviz
and how it interacts with ROS applications.
We developed ARviz using the Unity Game Engine. ARviz
is designed to work with many different plugins, which can be
categorized into two types: Display Plugins, which are used
for visualizing data received from ROS applications, and Tool
Plugins, which enable the use of natural input methods such
as speech and gestures for interacting with ROS applications.
The visual elements (holograms) generated by the plugins are
accurately localized in the real world with the help of Vuforia
Engine (https://www.ptc.com/en/products/vuforia/vuforia
-engine), a Unity tool kit that enables the alignment of the virtual
world and the real world, and Unity tf Listener, a component
of ARviz that collects spatial data (transformations
between coordinate frames) from the ROS application. ARviz
provides a hand menu, which can be accessed with a raise of
hand, as a convenient means to manage the plugins available
on the platform (Figure 3). The communication channel
between ARviz and the ROS application is established using
the Unity package ROS# (https://github.com/siemens/
ros-sharp) on the ARviz side and an ROS package rosbridge
(http://wiki.ros.org/rosbridge_suite) running on top of the
ROS application. We explain each component in more detail
in the following sections.
Figure 3. Hand Menu appears when the user raises their left
hand in front of the HoloLens 2. Hand Menu provides quick
access to the plugins available on the platform.
60 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
ARviz Plugins
Display Plugins are components of the ARviz platform that
allow users to visualize data from ROS applications in AR by
rendering virtual displays that are contextually located in the
real world. Display Plugins can be used for visualizing application
information such as robot transformation frames, navigation
paths, and robot gripper grasp poses (more details in
the " Use Cases " section). The plugins are designed to be reusable
for any ROS application using the same standard ROS
message types, and users can also implement additional customized
plugins to visualize other ROS message types or customized
ROS message types.
Data from ROS is collected by ROS# through a component
call a subscriber (more details in the " Communication
with ROS " section). The Display Plugin accesses the
Holograms Localization
Unity tf
Listener
Speech,
Gesture,
etc.
Visualization
Output
C#
Objects
rosbridge
ROS
Messages
https://www.ptc.com/en/products/vuforia/vuforia-engine https://www.ptc.com/en/products/vuforia/vuforia-engine https://www.github.com/siemens/ros-sharp https://www.github.com/siemens/ros-sharp http://wiki.ros.org/rosbridge_suite
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
