IEEE Robotics & Automation Magazine - March 2022 - 15

for initial alignment with the real world. During development,
the use of the robot base alone without it being coupled
with the table was also trialed. However, the table was kept
because it is visually more convenient to align a larger object,
and because the table was later used for robot-reconfiguration
guidance, once an ideal distal link transformation is generated.
On the other hand, the choice of the lectern in place of a
" tag-along " floating canvas was made, even though the latter
was seen more commonly in AR applications, and the user
employed voice commands to advance to the next task or
return to the previous task. This choice was made because the
canvas often gets in the way of the user and interferes with his
or her work, while having a fixed canvas located at a comfortable
position and orientation is more ergonomic. Both of the
hologram modules were defined with respect to frame {B}.
Following the initial alignment, the user interacts almost
exclusively with the contents on the lectern and the physical
robot, with the exception of placing the desired end-effector
hologram, which is grouped under the base module. The lectern
module consists of a lectern hologram with a dynamic
instruction panel overlaid on top. The user works through the
instructions and provides the system feedback via button presses.
The lectern design was chosen as it is ergonomically
designed for the user to interact with as they are standing. Its
slanted and large surface tilts the panel toward the user so he or
she can conveniently view its content without straining his or
her neck. In addition, it can be placed next to the working environment
and take over the job of a traditional 2D display. Conversely,
scene-content holograms and functional classes that are
related to the reconfiguration of the robot are grouped under
the robot's base module. This consists of the table, robot base,
desired end-effector model, reconfiguration workspace, and
postreconfiguration robot workspace unique to its topology.
Following the initial alignment of the base and the table
[Figure 6(a)], the user instructs the robot base to " anchor "
via voice commands, fixing the base in place. The anchoring
process also places the scene origin as a reference point at the
center of the hologram. The instructions that the user then
follows are in line with the reconfiguration workflow presented
in Figure 7 and can be summarized as follows.
First, the user starts a new configuration [see Figure 6(a)].
Then the user enters task 1, a scene consisting of the reconfiguration
space computed in the " Augmented Reality System
Design and Development " section. From the lectern shown
in Figure 6(b), the user can instantiate the hologram of the
distal link, and then use gaze and gestures to position and
orientate it to a desired configuration.
In conjunction, an end-effector-checking algorithm was
implemented to confirm whether the holograms lie within the
configuration space [Figure 6(c) and (d)] and further prevents
points that construct a topology, which is physically unachievable.
On the user's side, following a press of the " check " button,
the end-effector hologram will change its color depending on
its current state. A " green " color indicates that it is within the
workspace, and a " red " color suggests the opposite. Once the
hologram turns to green, a " ready " button is activated, and the
user is prompted to proceed [see Figure 6(d)]. The user then
confirms the coordinates of distal-link points P5 and 6, and
the system saves the configuration presented in the scene [see
Figure 6(e)]; this records the transformation of all the instantiated
holograms, closes the app, and sends the information to a
workstation. The workstation is responsible for computing the
optimal topologies of the malleable robot, defined by distance
geometry using a sampling of dihedral angles between the triangles
formed by the desired point and orientation of the end
effector and the fixed robot topology [7], which are used to
return mesh models of the achievable robot workspaces that
obtains the desired end-effector position and orientation as
well as the robot's topology transformations, represented by a
mesh model of the distal rigid link and distal joint in the new
optimal topology configurations.
Scene saving occurs after the user correctly places their
desired end effector in the reconfiguration space, but before
the workstation returns the mesh models of achievable workspaces,
along with their end-effector robot topologies. The
design decision of using scene saving and loading functions
over TCP/IP communication between the workbench and the
HoloLens was contemplated during development. The implementation
of sockets drastically reduces the frames per second
on the HoloLens and negatively impacts user experience;
however, it ensures a seamless transition between the initial
points-positioning stage and the reconfiguration stage of the
workflow, whereas a traditional saving and loading introduces
a break point. Ultimately, we prioritized user experience and
speed of reconfiguration over ensuring a seamless transition
as a jittery scene greatly diminishes the immersive experience
that an AR system should provide.
Reconfiguration and Alignment
To complete the necessary elements needed to define a reconfiguration,
the new robot's workspace and end-effector models
are loaded into the system [Figure 6(g) and (h)]. Note that
the workspace defined here differs from the reconfiguration
workspace as it is the volume that the robot can reach given a
particular end-effector reconfiguration, assuming that the
malleable link is rigid. In other words, the robot's workspace is
a subset of the reconfiguration workspace, defined when the
malleable link is rigid and the end effector is placed at a
unique position and orientation.
The system requests that the user provide a final confirmation
of the configuration scene [Figure 6(i)]. This is done to
ensure that the scene has successfully loaded the configuration
consisting of the desired end-effector pose, and that a robot's
workspace, which successfully encapsulates the end effector,
was generated. If any of these conditions are not met, the user is
free to reject the configuration and restart the process. Content
with the generated workspace, a mesh of the distal link, positioned
in the new robot configuration, is shown in the virtual
environment [Figure 6(j)]. This visualizes to the user exactly
where to reconfigure and position the malleable robot.
Finally, the user is walked through the reconfiguration
process via the lectern's instructions. During this stage,
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
15
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
