IEEE Robotics & Automation Magazine - March 2022 - 41

definitions for the categories of metrics used in the VAM-HRI
contributions and examples from the contributions on how
they implemented that metric for their application.
The following four objective metrics were used in the
VAM-HRI contributions:
●
Task accuracy: This is the proportion of correct predictions
to the total number of predictions (e.g., in Higgins et al. [9],
task accuracy is measured by the robot's ability to correctly
classify the objects referred to by the human).
●
Amount of training data: This is the amount of training
data that are collected or required for a machine learning
application (e.g., in Higgins et al. [9], the amount of training
data refers to the amount necessary to close the sim2real
gap versus learning in reality).
●
Task completion time: This is the amount of time between
tasks or events (e.g., in [14], it is the recorded time between
robot signaling and human reaction).
●
Task completion rate: This is the proportion of successful
attempts at a task to the total number of attempts at the
task (e.g., in [14], it is the number of successful completions
of a minigame in a VR robot game environment).
There were six subjective metrics used in the VAM-HRI
contributions:
●
NASA task load index (TLX) [8]: This is a multidimensional
scale for measuring user workload during and after task
execution (e.g., in Boateng and Zhang [2], it measures the
user workload of situational awareness in proximal
human-robot teaming with virtual shadows).
●
Perceived robot identification: This refers to the user's perceived
estimates about the robots in the environment (e.g.,
in Boateng and Zhang [2], users identified the position,
orientation, and movement patterns of an out-of-sight
robot member based on virtual shadows).
●
System Usability Scale (SUS) [3]: This questionnaire measures
a user's perceived usability of a system (fitness for
purpose) on a seven-point Likert scale ranging from
" strongly disagree " to " strongly agree " (e.g., in Ikeda and
Szafir [10], the SUS is used to assess the AR robot debugging
tool's usability).
●
Think-out-loud process: In this technique, participants
actively voice their thoughts when using an application for
researchers to receive real-time feedback (e.g., in Ikeda and
Szafir [10], participants talk out loud about their thought
process when using the AR robot debugging tool).
●
Interviews: Researchers ask participants to comment on
specific features after using the VAM-HRI applications
(e.g., in Wadgaonkar et al. [22], they request participants to
comment on which robot features, such as color and texture,
impact robot behavioral anthropomorphism in VR).
●
Custom survey questions: These surveys are similar to interviews,
except that users fill out specific custom survey
questions that are application and task specific (e.g., in Higgins
et al. [9], users are asked about what they found frustrating
for training ground language models in VR with
simulated robots; in Mimnaugh et al. [17], users reported
on VR sickness).
Current Trends and the Future of VAM-HRI
In this article, the Fourth International VAM-HRI Workshop
is used as a case study for MRIDE classification and categorization
within the reality virtuality interaction cube; however,
the papers submitted to this workshop can also be used to
exemplify and project current and future trends in the field of
VAM-HRI. This growing subfield of HRI is showing promise
in enhancing all areas of HRI from robot control (e.g., teleoperation
and supervision interfaces) to collaborative robotics
and improving teamwork with autonomous systems. The following
will cover some of the key insights gathered from this
year's workshop that show how VAM-HRI is evolving and
improving the field of HRI as a whole.
An Experimental Evaluation of VAM-HRI Systems
Research in HRI heavily features user studies in the evaluation
of robotic systems and their interfaces. It has been an
ongoing challenge to adequately record and play back human
interactions with robots
to answer questions such
as: " Where was the user
looking at X time?, " " How
close was the human positioned
relative to the robot
at Y moment?, " and " What
were the user's joint values
when using a new interface,
and how are the physical
ergonomics evaluated? "
As a possible solution to
many of these challenges,
VAM-HRI allows for
unprecedented recording,
playback, and analysis of
user interactions with virtual
or real robots and
objects in an experimental
setting because of the
inherent ability of HMDs
(and other devices like a Leap Motion) to record body/hand/
head position/orientation and gaze direction from a seemingly
limitless number of virtual cameras recording from different
angles [24]. This is exemplified at a highly polished level
in CoBot Studio [14] (see Figure 3).
However, it is interesting to note that, although precise
The CoBot Studio project
unites roboticists,
psychologists, artificial
intelligence experts,
multimodal communication
researchers, VR developers,
and professionals in
interaction and game
design.
objective measures can be relatively easily gathered from
VAM-HRI experiments, only two of the 10 submissions to the
fourth VAM-HRI workshop gathered any objective data (see
Table 2). The lack of objective measures may be due to a
handful of factors, such as the work being in a preliminary
stage best suited for a workshop or the research questions
being more focused on social responses and subjective opinions
from users. Regardless of the reason, we encourage
authors of future VAM-HRI submissions to any venue to take
full advantage of the objective measurements that VAM-HRI
systems inherently provide as objective observations are still
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
41
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
