IEEE Robotics & Automation Magazine - March 2022 - 36

To showcase the TOKCS's
capability, it is applied
to the 10 papers from
the Fourth International
Workshop on VAM-HRI and
examined for key trends
and takeaways.
Background
The need to help identify growing trends within VAM-HRI is
evidenced by four consecutive years of a VAM-HRI workshop
consistently spanning 60-100+ attendees. This nascent subfield
of HRI addresses challenges in mixed reality (MR) interactions
between humans and robots, involving applications
such as remote teleoperation, mental model alignment for
effective partnering, facilitating
robot learning, and
comparing the capabilities
and perceptions of
robots and virtual agents.
VAM-HRI research is
becoming even more
accessible to the robotics
community, due in part to
the widespread availability
of commercial virtual reality
(VR), augmented reality
(AR), and MR platforms
and the rise of readily accessible
3D game engines for
supporting virtual environment
interactions.
To understand what challenges and solutions have been
emphasized by this new community, Williams et al. [25] proposed
the reality-virtuality interaction cube as a tool for clustering
VAM-HRI research. The interaction cube is a 3D
conceptual framework that captures characteristics about the
design elements involved [expressivity of view (EV) and flexibility
of control (FC)] as well as the virtuality they implement
(from real to fully virtual). While the interaction cube provides
a useful lens for roughly characterizing research involving
interactive technologies within VAM-HRI, the continuous
nature of the cube makes it challenging to exactly position
where design elements and environments are within it.
Furthermore, the interaction cube does not address other
characteristics of VAM-HRI research that have recently
gained attention, such as robot internal models, software,
hardware, and experimental evaluation methods.
To help advance the understanding of different VAM-HRI
systems, we introduce the TOKCS. The TOKCS builds off
work from the interaction cube, discretizing its continuous
scales and adding new key characteristics for classification.
The tool is applied to the 10 workshop papers from the
Fourth International Workshop on VAM-HRI to validate its
usefulness within the growing subfield. These classifications
help inform current and future trends found within the workshop
and VAM-HRI as a whole.
The Interaction Cube Framework
The interaction cube [25] uses three dimensions to characterize
VAM-HRI work: the 2D plane of interaction to represent
interactive design elements and the 1D reality-virtuality continuum
of Milgram et al. [15] to characterize the environment.
Interaction Design Elements:
Enhancing View and Control
The first two dimensions of the interaction cube (Figure 1) are
defined by the plane of interaction, which captures both 1) the
opportunities to view the robot's internal model and 2) the
degree of control the human has over the internal model. These
two levels of interactivity (termed the EV and FC, respectively)
are the conceptual pillars for characterizing interactivity within
the interaction cube, and any components that contribute or
impact either EV or FC are called interaction design elements.
This is similar to the model-view-controller design pattern.
However, in this case, the 2D placement on the interaction plane
depends on a vector whose direction results from the impact a
design element has on EV and the impact a design element has
on FC. The magnitude of the vector is scaled by the complexity
of the robot's internal model. According to Williams et al. [25],
" while it is likely infeasible to explicitly determine the position of
a technology on this plane, it is nevertheless instructive to consider
the formal relationship between interaction design elements
and the position of a technology on this plane. "
Manipulable Artifacts in VR
1
Teleoperation in VR
0.5
Holographic Control Panel
0.2
0.4
EV
0.6
0.8
1 0
●
Figure 1. The reality-virtuality interaction cube used to visually
categorize MRIDEs according to their FC and EV and where they
lie upon the reality-virtuality continuum. Reality is indicated as 0
and virtuality as 1. FC: flexibility of control; EV: expressivity of view.
36 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
0.5
Manipulate Plan
Artifacts in AR
1
MR Interaction Design Elements:
Anchoring and Artifacts
The interaction cube categorizes the study of VAM virtual
objects as MR interaction design elements (MRIDEs), which
can fall into one of three categories:
●
User-anchored interface elements: These are objects
attached to a user view, similar to traditional GUI elements
that are anchored to the user's camera coordinate frame
and do not change along with the user's field of view. These
elements may also be referred to as part of a user's headsup
display as popularized by video games and movies.
Environment-anchored interface elements: These objects are
anchored to the environment or a robot, for example, virtual
arms that can be anchored to a robot [7] or virtual
objects that can be anchored to the physical environment.
FC
Reality-Virtuality
Continuum
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
