IEEE Robotics & Automation Magazine - March 2022 - 37

●
Virtual artifacts: These are objects that can be manipulated
by humans or robots or may move " under their own ostensible
volition " [25]. For example, virtual indicators of robot
position, such as arrows, can move on their own within the
environment.
The Reality-Virtuality Continuum and VAM-HRI
The third axis of the reality-virtuality interaction cube illustrates
where an MRIDE falls on the reality-virtuality continuum
[15]. This continuum classifies environments and interfaces
with respect to how much virtual and/or real content they contain.
On one end of the spectrum lies reality, which is any interface
that does not use any virtual content and makes use of only
real objects and imagery. The opposite end of the spectrum is
VR, which would be an interface that consists of pure virtual
content without any integration of the real world (for example,
a simulated world presented in VR). Between these two
extremes is MR, which captures all interfaces that incorporate a
portion of both reality and virtuality in their design. There are
two subclasses of MR: 1) AR, where virtual objects are integrated
into the real world, and 2) augmented virtuality (AV), where
real objects are inserted within virtual environments.
AR interfaces in VAM-HRI often communicate the state and/
or intentions of a real robot. For example, the battery levels of a
robot can be displayed with a virtual object that hovers over a real
robot, or a robot's planned trajectory can be drawn on the floor
with a virtual line to indicate its future movement intentions.
VR interfaces are often used to provide simulated environments
where human users can interact with virtual robots. In
these virtual settings, user interactions with robots can be
monitored and evaluated without risk of physical harm for
either robot or human. Additionally, the virtual robot models
can be easily and quickly altered to allow for rapid prototyping
of both robot and interface design. Without the need for
physical hardware, robots can be added to any virtual scene
without the typical costs associated with real robots.
Virtual environments can also be used to teleoperate and/
or supervise real robots in the physical world. In cases like
these, 3D data collected by the real robot about its surrounding
environment are integrated within virtual settings to create
AV interfaces. Cyberphysical interfaces and virtual control
rooms are two common VAM-HRI AV methods of enhancing
a remote robot operator's ability by increasing situational
awareness of the robot's state and location while mitigating
the limitations of virtual interfaces, such as cybersickness [13].
The TOKCS Classification Framework
The key insight of this work is the addition of key characteristics
of VAM-HRI not covered by the interaction cube to create
a TOKCS. These include VAM-HRI system hardware,
research that seeks to increase the robot's model of the world
around it, and additional granularity to MRIDEs. The characteristics
are part of the TOKCS, which is then applied to the
fourth VAM-HRI workshop papers in the section " Paper
Classifications of the Fourth VAM-HRI Workshop. " The
application informs
recommendations outlined in the section " Current Trends
and the Future of VAM-HRI. "
Hardware
While hardware used for
VAM can vary widely,
there are certain types of
hardware that are commonly
used in VAM-HRI.
Here we outline the most
common, which enable
experiences along the
reality-virtuality continuum:
head-mounted displays
(HMDs), projectors,
displays, and peripherals.
Because hardware technology
is making significant
advances every year,
labeling the specific technology
(e.g., HoloLens 2)
is important when classifying
hardware within the
TOKCS. These hardware
technologies then fall
under these categories.
●
A key property of virtual
object manipulation is the
user's action attribution
of the manipulation (i.e.,
whether the user perceives
that he/she moved the
object, the robot moved the
object, or the object moved
on its own).
HMDs: VR, MR, and AR all commonly use HMDs. Oculus
Quest and HTC Vive both allow for a full VR experience,
visually immersing the user in a completely virtual
environment. HTC Vive also allows for AV, such as in
Wadgaonkar et al. [22], where the user is in a virtual setting
but the virtual robot being manipulated is also moving in
the real world. The Microsoft HoloLens and the Magic
Leap are strictly AR headsets, where virtual images are rendered
on top of the real-world view of the user.
●
Projectors: Onboard projectors can provide a way for the
robot itself to display virtual objects or information. Alternately,
static projectors allow an area to contain AR elements.
Images might be projected onto an object, the floor,
or a robot.
●
Displays: This category of hardware ranges from handheld
smartphones or tablets to room-size displays. 2D and 3D
monitors fall somewhere in between this range. Some of
them exist in a single location, while mobile displays can be
carried by a person or moved by a robot. A cave automatic
virtual environment immerses the user in VR using three
to six walls to partially or fully enclose the space. An AR
display might include a real-time camera with overlaid virtual
graphics, while a VR display contains completely virtual
graphics. Displays can be an especially effective way to
conduct user studies without investing in expensive hardware,
for example, by showing recorded videos to participants
on Amazon Mechanical Turk (MTurk) [18].
●
the insights and future work
Peripherals: Peripheral devices allow for a richer interaction
within VR, AR, or MR. Leap Motion hand tracking
can be combined with a headset, such as the HTC Vive
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
37
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
