IEEE Robotics & Automation Magazine - March 2022 - 38

(as in [14]), to provide recording and playback of motions
and commands. Oculus Quest controllers are handheld
and can be used individually or in tandem, giving the
user a modality for both gesturing and selecting with the
use of buttons on the device. Peripherals may frequently
be used to enhance the
FC of an MRIDE.
An important component
of VAM-HRI research
programs is to evaluate and
benchmark new approaches
by using both objective and
subjective metrics.
Software
There is an assortment
of software applications
for facilitating 3D environments
for VAM-HRI
research. The most popular
platforms, like Unity3D,
support a wide variety of
VR and MR hardware, like
those outlined in the section
" Hardware, " and offer
packages for networking
with robot networks like Robotic Operating System (ROS)
servers and rendering robot sensor data. ROS also offers a
robot simulator, Gazebo, that directly interfaces with ROS
applications and has been used for VAM-HRI research. Other
additional software generally relevant to HRI research is also
included here, such as tracking AR tags to detect object poses
using TagUp [1]. Software is not a direct part of the
(a)
interaction as hardware, but we report relevant software for a
holistic understanding of the resources that the VAM-HRI
community uses to develop their applications.
Robot Internal Complexity of Model
The interaction cube emphasizes the increased EV and FC
aspects of projected visual objects on the robot's underlying
model. This fails to explore, however, the sensing capabilities
and data afforded by VAM technologies [e.g., augmented
reality head mounted display (ARMHD)]. The framework
can be expanded by including the technologies' ability to aid
the robot's internal model of the world-namely, increasing
the robot's internal complexity of model (CM). The robot's
internal CM benefits from data typically difficult to gather
(e.g., eye gaze) as well as the technology affording data
assumptions (e.g., a headset with various sensors anchored to
the user's head). These data manifest in aiding a robot's model
of the environment and/or model of the user.
●
Environment: Data from the VAM technology further
increase the robot's understanding of an environment. An
example is provided in Figure 2. Given a mobile robot with
2D SLAM, a 3D map from an ARHMD's SLAM can be
transformed into the robot's coordinate frame. The map
can then be used for more accurate navigation. In another
situation, a mobile phone camera can help with object recognition,
both in front of and behind the robot.
●
User: Data from VAM technology further increase the
robot's understanding of the user. For example, a robot
can better infer a user's intent to choose an object by
using ARHMD eye gaze [20]. Data gathered from motion
sensors can be used for both functional purposes (e.g., to
determine where the human is in relation to the robot) as
well as to infer an affective human state, such as student
curiosity [6].
User-Perceived Anchor Locations
(b)
Figure 2. A demonstration of a navigation situation where (b)
the robot 2D SLAM map benefits from (a) the 3D SLAM map
from the ARHMD. The robot only maps the two front table legs
(bottom left) as it is only equipped with a 2D lidar. The robot,
however, is too tall to move past the table, so it will collide if it
does not use the 3D map from the ARHMD. A combined SLAM
map would be created from feature matching such as the table
legs (circles).
38 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
and Manipulability
The MRIDE categorizations of user-anchored interface elements,
environment-anchored interface elements, and virtual
artifacts (described in the section " MR Interaction Design
Elements: Anchoring and Artifacts " ) are not mutually exclusive
and lack the necessary granularity. For example, a virtual
artifact can be user anchored, such as a movable useranchored
element or an environment-anchored object that
moves on its own. Granularity can also be added to benefit
MRIDE classifications, such as distinguishing between robotand
environment-anchored objects.
To this end, two important distinctions can be added to
expand the current framework. First, we apply two characteristics:
Anchor Location {User, Robot, Environment} and Perceived
Manipulability {User, Robot, None}. Second, we
distinguish MRIDEs based on the intended user perception
of the virtual object (i.e., where the user perceives the anchor
to be and who can/does move a virtual object).
The first distinction allows for multiple labels within each
characteristic, such as objects that are manipulable by both the
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
