IEEE Robotics & Automation Magazine - March 2022 - 14

position defined by the four distances, and using it to infer the
positions of P3-5 (which is trivial as the relative position of
P3-5 does not change with respect to the distal link). To
obtain this, first the new locations of P3 and 4 were determined
using the distances. The mesh model is defined with
P3 at [0, 0, 0], with P4 directly along the z-axis. We can then
compute the translation of the mesh model based on the
new position of P3, and then we can rotate the model
around P3 to align to the new P4 using a rotation matrix.
The rotation matrix is computed using the initial model
([ ,, ])00 50
p34model
=
and new-world ()PP
p34 43world
=vectors
that define the orientation of the mesh model.
1,200
1,000
800
600
400
200
-200
-400
-600
-1,000
-500
P3/4 Experimental
P5 Theoretical
P5 Experimental
X (mm)
Robot-Mounting Platform
P3/4 Theoretical
Achievable P5 Theoretical
Figure 4. Results of the reconfiguration space exploration,
shown for the end of the malleable link (P3/4 experimental)
and the robot's end effector (P5 experimental). The theoretical
reconfiguration space is overlaid for both, along with the
positioning of the robot mounting platform.
500
1,000
Coordinate System and Initial Alignment
To combine the reconfiguration space and the optimal solution
visualization algorithm into an HRI platform, an appropriate
AR interface was developed to enhance the user
experience in manipulating malleable robots. Prior to this,
this section introduces the reference frames attached to each
holographic object in the AR space as such relations lay the
foundations of an accurate alignment.
The ultimate goal of understanding scene-coordinate systems
is to infer the location of the end effector with respect to
the base. The AR environment consists of four coordinate
systems {H}, {E}, {B}, and {W}, illustrated in Figure 5, and
corresponding to the HoloLens, end effector, base, and world
origin, respectively. Using its four tracking cameras, the HoloLens
is able to infer its own transformation {H} with respect
to a stationary world origin {W}. It is also able to resolve spatial
relationships between virtual objects.
Before proceeding with the AR application reconfiguration,
the virtual and real environments must be aligned at the
base {B} of the robot. This can be performed automatically,
with the addition of error-metric feedback; however, due to
the computational limits of the HoloLens, this was performed
manually by the user, allowing the HoloLens to focus
entirely on performing at its maximum rendering frequency.
During the initial alignment phase, the table and robot base
holograms are freely movable in terms of position, but with
its orientation in x and y fixed. This was done to improve the
comfort of the alignment process, but more importantly it
limits the alignment error and its effects on the system.
By default, the Unity game engine environment used for
the development of this app defines all the holographic
objects with respect to {W}, hence, mathematically, it needs to
be transformed into a frame {B}. For example, to find
ET ,
B
E
and the base TW
B
the
transformation from the base to the end effector, given the
relationship between the end effector TW
with the origin:
with the origin,
ETT .T
B
=
{H}
{E}
{B}
W
E
-
W
1 B
(5)
Under Unity, this procedure can be accomplished by
defining the local position of virtual objects with respect to
the base frame {B} using a parent-child relationship.
AR Interface
We begin by defining the tools that are necessary to perform a
pose reconfiguration in the context of malleable robots. This
includes all the information needed to define the desired endeffector
position and orientation as well as the postreconfiguration
robot workspace.
The virtual world consists of two main hologram modules:
{W}
Figure 5. The coordinate systems of the scene. W: world origin;
B: robot base; E: end effector; H: HoloLens.
14 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
the robot base and the lectern. Each module consists of its
own suite of peripheral holograms, which are designed to
assist the user and provide instructions. The idea behind
choosing these two objects as the main HRI medium is simple.
For the robot base, as it is an integral part to the robot, it
is perfectly logical that it is displayed as a hologram to allow
Y (mm)
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
