IEEE Robotics & Automation Magazine - March 2022 - 12

with some users given no AR support, some given 2D AR
visualization, and the remaining given HMD AR support.
The results indicated that the users of HMD yielded better
accuracy ratings (75%) compared to 2D AR users (65%). One
reason behind this was that having a 3D holographic object
mitigates errors caused by the user trying to map a 2D projection
of 3D objects onto their vision.
The potential of AR-/HMD-driven robotic systems do not
stop there. In Bolano et al.'s work [19], they used 3D cameras
and point clouds to predict robot collision via trajectory and
swept volume visualization. What is interesting in this work
was the ability of their system to project dynamic robot workspaces
to AR such that it can enhance user awareness and
enable path replanning. Robot reprogramming was also seen in
Kousi et al.'s work [20], where markerless, AR-based HRI was
used to aid the user with controlling a mobile robot in a production
environment in the case of an unexpected error. Virtual
buttons were implemented to initiate robot movements, and
Microsoft
HoloLens
Malleable
Robot
OptiTrack
Tracking
System
the robot provided real-time information on active tasks at the
workstations. In summary, AR in HMDs is a promising technology
that is worth exploiting for enhancing HRI.
AR System Design and Development
The AR application designed in this article was for a malleable
robot system presented in [7] and preliminarily discussed in
[6]. It is a 2-DoF, reconfigurable robot formed of a vertical,
revolute joint at the base, with a malleable link connecting this
base to a second revolute joint, and a rigid link connecting the
second revolute joint to the end effector. This is an extrinsic,
malleable robot, and thus we believe that there is potential for
an AR system to recommend the end-effector placement.
We deployed our AR system on the first generation of
Workbench
Hologram
Distal Link
Hologram
Lectern
Hologram
Microsoft HoloLens. AR/MR was selected over VR as it
allows for direct manipulation and co-located visualization of
the malleable robot reconfiguration. The HoloLens was
selected because it possesses many sensors that capture environmental
information and the user's gestures by using its
four environment-understanding cameras and one depth
camera. All the applications for the HoloLens were developed
on the Unity game engine (Version 2018.4.27f1) and the
MRTK (Version 2017.4.3.0). As the HoloLens behaves similarly
to a mobile device, no dedicated PC is needed to run the
applications, except for loading the application in the first
place. However, the developed system does require an additional
workstation to compute, then generate, the robot's
workspace as well as the desired end-effector positions. OptiTrack
Motive (Corvallis, Oregon, USA) software was also run
using this workstation to perform the motion tracking. The
specifications of the workstation were a Ryzen 9 5900X CPU
and 64-GB random-access memory (RAM).
Figure 1. The developed AR-assisted reconfiguration system
of a malleable robot. The user is shown interacting with the
AR-assisted points-placement scene overlaid on top of the robot
base, with the presence of a transparent configuration space.
The lectern to the right prints out the instructions for the task the
user needs to conduct.
P4
P6
P3
P2
P1
Figure 2. The previously developed 2-DoF malleable robot to
be reconfigured, highlighting the point ()P15representation
and
interpoint distances (D13, D14, D23, D24) that define the
malleable-link topology.
12 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
P5
The Malleable Robot-Reconfiguration Workspace
Given the desired end-effector position and orientation, a set
of optimal robot topologies (that is, specific reconfigurations)
can be generated that achieve this [7]. With that said, the
developed system should allow for efficient reconfiguration
by classifying whether the user's demands are valid, demonstrating
visually what is achievable, and feeding this information
back to the user through an AR-enabled interface (see
Figure 1). Ultimately, this implies robustly defining a reconfiguration
workspace, defined as the volume in which all points
within this space can be reached by the malleable robot through
reconfiguration (not through joint movement) of the robot's end
effector when it is not fixed in a certain topology.
For the developed AR application, it was necessary to
compute the reconfiguration workspace for the end effector
(P5, as shown in Figure 2), aiding the user in ensuring that
the position of the desired end effector is within it. However,
this is difficult due to the variability in the malleable link
(bend, extension, compression, and twist).
To simplify the determination of the reconfiguration
workspace, a twist of the malleable link was not considered.
Using the minimum radius of curvature (),
rmin
the involute
and cycloid curves of the malleable link at its maximum
D24
D14
D13
D23
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
