IEEE Robotics & Automation Magazine - March 2022 - 62

Unity tf Listener
Data in ROS are specified with respect to a given base frame
to provide physical spatial context (e.g., a robot's coordinate
may be specified relative to the map frame). ROS is a distributed
computing system by design. Hence, in ROS applications,
transformations
between different coordinate
frames are asynchronously
broadcasted
(published) by different
processes (nodes). The
ROS component named
tf Listener collects and
assembles all of this
transformation information
and allows any
component of the ROS
application to query
The tf Display Plugin visualizes coordinate frames moniThe
holograms are
accurately localized in the
real world with the help
of Vuforia Engine and
Unity tf Listener.
transformation between any two coordinate frames. The
Unity tf Listener provides the analogous functionality in
ARviz. Asynchronous transformation information from the
ROS application is passed on to the Unity tf Listener
through ROS#. The Unity tf Listener collects and assembles
the transformation information. Display Plugins and Tool
Plugins can then query transformation between any two
coordinate frames and render display elements at appropriate
locations.
Communication with ROS
For the plugins to receive and send data to and from ROS
applications, a connection with the ROS environment is
required. ROS runs on Ubuntu, and within ROS applications,
nodes (i.e, processes) communicate with each other
by broadcasting messages (i.e., data structures) over topics
(i.e., communication channels). Publishers are used to
broadcast messages and subscribers are used to receive and
manage messages. On the other hand, ARviz is built on
the Unity Game Engine, a platform that does not natively
support ROS APIs for establishing the communication
channels mentioned earlier. To resolve this problem, a
combination of rosbridge on ROS and ROS# on Unity was
used. rosbridge and ROS# allow ARviz and any ROS application
to communicate via a common standard JSON
string format over TCP/IP, as shown in Figure 2. rosbridge
establishes the network port for ARviz to connect to
ROS# provides ROS-like APIs such as ROS# Publisher and
ROS# Subscriber, enabling ARviz plugins to communicate
with other ROS applications through ROS-like communication
mechanisms.
Available Plugins
Three Display Plugins and two Tool Plugins were implemented
in our initial release of ARviz. These plugins are developed
and tested on the AR-head-mounted-display Microsoft HoloLens
2 (https://www.microsoft.com/en-AU/hololens/hardware)
and are summarized in Figure 5.
62 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
tored by the Unity tf Listener. Each coordinate frame is visualized
as a set of 3D axes, with red, green, and blue
corresponding to x, y and z axis. Figure 6(a) shows examples
of visualizing the joint/link frames of two robots. The plugin
can also show the name of each frame and the connection
between it and its parent frame, depicted with an arrow. The
visibility of each element of this Display Plugin can be toggled
through a control panel as shown in Figure 6(b). The tf
Display Plugin can be used for visually confirming the connectivity
between ARviz and the ROS application as well as
verifying the alignment between the VWCS and the RWVS.
Moreover, tf Display Plugin allows users to visualize many
aspects of robot operation, including robot localization and
robot structure, and serves as a powerful development and
debugging tool. Users can utilize tf Display Plugin to view
the connections between the coordinate frames within the
robot for a better understanding of how the robot base, its
multilink manipulator and other degrees of freedom, operate
as a whole.
The VisualizationMarkerArray Display Plugin visualizes a
collection of user-defined markers. These markers include
various primitives such as cubes and spheres, text, or lines.
This plugin is inspired by the MarkerArray (http://wiki.ros
.org/rviz/DisplayTypes/Marker) display in Rviz and provides
the visualization capabilities similar to its rviz counterpart.
The VisualizationMarkerArray display plugin allows users to
display arbitrary primitives in the virtual world through a
user-defined ROS topic by publishing messages of type visualization_msgs/MarkerArray
to this topic. This plugin provides
a generic tool for creating visual markers with user-defined
size, color, shape, and location. This tool can be used for a
wide range of tasks such as creating text labels, visualizing
robot paths, and generating object indicators.
The Stamped Pose Display Plugin visualizes the translation
and orientation of a virtual object with respect to a given
coordinate frame, given in the format of a geometry_msgs/
PoseStamped ROS message type. The default virtual object
used for the visualization is an arrow, but users can customize
this plugin by defining their own preloaded object model or
mesh to be displayed instead the default arrow. The plugin listens
for any incoming messages being published to a topic
defined by the users and visualizes the pose with the virtual
object accordingly. This plugin is useful for visualizing the 3D
pose of robots or occluded objects, for example.
The 2D Arrow Tool Plugin allows users to specify a 2D
pose (position and orientation) by placing an arrow on the
ground plane in AR through the use of hand gestures. When
the user raises their hand, a ray emanating from the hand is
displayed in AR as a dotted line. When the user taps the index
finger with the thumb (known as an " air tap " gesture), the
location where the ray intersects the ground plane is identify,
and an arrow is placed with its tail endpoint fixed at this location.
Next, the user can rotate the arrow on the floor by moving
the raised hand-the arrow tip will follow the intersecting
point between the hand ray and the ground plane. By
http://wiki.ros.org/rviz/DisplayTypes/Marker http://wiki.ros.org/rviz/DisplayTypes/Marker https://www.microsoft.com/en-AU/hololens/hardware
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
