IEEE Robotics & Automation Magazine - March 2022 - 11

accuracy of the system in terms of initial calibration and
overall accuracy, and demonstrate its viability. The results
demonstrated a good performance, with an average
repositioning accuracy of 9.64 ± 2.06 mm and an average base
alignment accuracy of 10.54 ± 4.32 mm in an environment the
size of 2,000 mm × 2,000 mm × 1,200 mm.
Introduction
The difficulty of designing, trajectory planning, and orientation
control of a multi-DoF robot are common problems
faced by researchers in task-oriented robot design [1].
Although these issues have been mitigated as robotics continues
to evolve, with better computational tools developed to
control manipulators with higher accuracy and repeatability
implemented [2], one aspect that remains challenging is the
development of solutions where a robot must complete tasks
in collaboration with a user, instead of operating fully autonomously.
Despite the progress made in the last decade with the
deployment of collaborative robots [3]-and in human-robot
interaction (HRI) research [4]-under an industrial setting,
robots still excel at performing precise, accurate, and repetitive
tasks while being kept away from humans as it is still
difficult to visualize their intentions in real time and communicate
that to a nearby user [5].
In this article, we introduce an AR-enabled reconfiguration
interface for serial, malleable robots, thus improving the
accuracy of end-effector positioning following a robot reconfiguration.
Malleable robots are a type of reconfigurable serial
robot developed by Clark and Rojas [6], [7], based around the
design of a malleable link-a variable-stiffness link between
the revolute joints of the robot that allows for their variable,
relative positioning. This results in a low-DoF robot having
significantly increased task versatility due to the variable
workspace of the robot. For extrinsic, malleable robots, where
their reconfiguration is performed externally by a user directly
reshaping the malleable link, one of the key issues that arises
is performing this alignment accurately. For example,
visualizing the new configuration of the robot is difficult for a
user and results in a limited positioning accuracy [7]. The
developed AR system enables the user to smoothly generate
and place a virtual end effector to within the maximum
reachable space that the robot can move (referred to as reconfiguration
space). Using this desired position and orientation,
the system can advise optimal end-effector transformations,
compute the workspace for each topology, and ultimately
overlay them in front of the user's eyes. This allows for easier
and more accurate alignment of the malleable robot to an
optimal position by the user.
The Significance of Extended Reality in HRI
Despite HRI being a well-studied field, many researchers
agree that there is a need for a shared understanding of a
robot's workspace to enhance collaboration and situational
awareness. Although this method of interaction is still novel, a
study by Solyman et al. [8] with 2D workspace visualization
has demonstrated the potential of these workflows to
accelerate robot-assembly tasks. However, the biggest drawback
of showing the workspace on a PC is that it requires the
user to constantly switch between looking at the display and
the robot, which can become tiresome. Furthermore, it can be
challenging for the user to map a 2D projection of an image
to the 3D world, leading to poor positioning accuracy and
increasing operation time. This problem is particularly critical
in medical applications, where the lack of information on
image depth may compromise patient safety due to incorrect
placement of visual markers [9]. Overall, these challenges
were commonly encountered in 2D workspace visualization
studies in literature, thus demonstrating the importance of a
3D immersive experience.
One solution to solve these issues is taking the surrounding
environment completely out of the problem via the use of
virtual reality (VR)-enabled teleoperation. Unfortunately, this
introduces the problem of mapping a user's reference frame
onto the robot's reference frame, where an inaccurate bidirectional
mapping leads to poor user experience [10]. With that
said, simpler solutions to that problem have been found within
the Baxter's Homonculus system developed by Lipton et al.
[11], which leverages off-the-shelf VR goggles to give the user
the same point of view as the in-action robot by introducing
an intermediate " VR control room " that takes control of the
mapping. This decouples the provision of sensory stimuli
directly from the robot to the user's VR scene and allows the
user to select which of their movements translates to the
robot. Although VR technology may provide a user-friendly
interface, it does not enable the user and the robot to collaborate
harmoniously within the same shared space. When evaluating
its accuracy, the Baxter's Homonculus study also
presents little insight into the degree of error in the mapping,
instead focusing on other practical aspects in manufacturing
such as time consumption of assembly and number of grasps.
Following the introduction of AR, image overlay of 3D
holograms onto the users vision was made possible. AR systems
have seen applications in a wide range of working scenarios,
fulfilling the purposes of motion planning [12], control
[13], and visualization [14]. The main benefits of having an
AR-assistive system was well summarized by Makhataeva and
Varol's review [15]. The studies examined in their work displayed
varying levels of accuracy, with some AR-enabled teleoperation
procedures displaying 60-80% accuracy when
trying to replicate motions with digital cameras [16]. The
other systems used in neurosurgical applications achieved
mean target errors as low as 1.11 ± 0.42 mm [17] by incorporating
fiducial markers and using intraoperative reimaging to
adjust for tissue shifting.
Recently, the growth in popularity of the Microsoft HoloLens
has attracted developers to enter the head-mounted displays
(HMDs) AR community. Together with the Mixed
Reality ToolKit (MRTK) [18], developers are able to create
interactions through methods such as gaze, speech, and gesturing
in their AR apps while preserving the benefits of AR.
An interesting study conducted by Rosen et al. asked users to
label whether collision occurs for a set of trajectories [12],
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
11
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
