IEEE Robotics & Automation Magazine - March 2022 - 40

Table 1. Summary of the TOKCS.
Paper
Anchor
Location
Boateng and Zhang [2] Robot, Env
Ikeda and Szafir [10]
LeMasurier et al. [11]
Puljiz et al. [19]
Env
Perceived
Milgram
Manipulability EV FC CM Continuum [15] Software
Unity
#
User
Env, Robot User
Wadgaonkar et al. [22] Env, Robot
Barentine et al. [1]
Env
Higgins et al. [9]
Mara et al. [14]
Mimnaugh et al. [17]
Mott et al. [18]
User
Env
User
Robot, User
Env, User
#
#
#
#
# #
#
# #
#
#
AR
AR
AV
AV
AV
VR
VR
VR
VR
VR
Unity
Unity
Unity
Hardware
HoloLens video
recordings via
MTurk
HoloLens
Unity, ROSNET, ROS HTC Vive
HoloLens
HTC Vive
Unity, TagUp
Unity, ROS#, ROS,
Gazebo
Unity
Unity
Unity
Oculus Quest
VR headset and
controllers
SteamVR headset
HTC Vive Pro Eye
and Leap Motion
Oculus Rift S
MTurk Web Video
of VR
Up arrow symbols (#) indicate that the work increases the functionality within this aspect of the TOKCS. Blank entries indicate that the contributions of
the paper for this aspect are on par with prior work. Env: environment
level. The anchors of two methods are located at the robot,
and two are located at the user. If a perceived manipulable is
available, it is typically available at the user level.
We also observe a broad range of utilized hardware and
software. Unity was overwhelmingly popular among papers
as the 3D game engine of choice; nine of the 10 papers explicitly
mention Unity3D. The most popular HMD mentioned
Table 2. A description of objective and subjective
metrics in the fourth VAM-HRI workshop papers.
Paper
Objective Metrics Subjective Metrics
Boateng and
Zhang [2]
Ikeda and
Szafir [10]
Wadgaonkar
et al. [22]
Higgins
et al. [9]
Mara
et al. [14]
Mimnaugh
et al. [17]
Mott et al. [18]
Task accuracy;
amount of
training data
Task completion
time; task
completion rate
Custom survey
questions
Custom survey questions
Blank spaces indicate a lack of metric of that type for that paper.
Papers omitted from the table did not report metrics. TLX: task load
index; SUS: System Usability Scale.
Custom survey
questions
NASA TLX; identification of
robot position, orientation,
and movement
SUS; think-out-loud
process
Post-experiment interviews;
custom survey questions
Custom survey
questions
was the HoloLens, which was used in three of the papers.
Oculus Quest, HTC Vive, and MTurk are each used in two of
the 10 papers.
Evaluations: Subjective and Objective Metrics
In addition to the TOKCS, we further evaluated measures and
metrics applied to VAM-HRI research. An important component
of VAM-HRI research programs is to evaluate and
benchmark new approaches by using both objective and subjective
metrics. Objective metrics are any metrics that can be
directly determined through sensors or measurements and do
not involve a human's subjective experience. Examples of
objective metrics include task completion time, the number
of successful and failed trials, and accuracy and precision of
visualization alignment.
Subjective metrics are any metrics that depend on the perceived
experience of the users involved. Examples of subjective
metrics include the mental workload, levels of
immersiveness, and perceived system usability. Both subjective
and objective metrics are important and complementary
benchmarks for determining how effective new VAM-HRI
contributions are compared to existing approaches. A wide
variety of metrics is available for these measurements, and
understanding which metrics VAM-HRI researchers are
using helps highlight what aspects of interaction these technologies
are improving.
The most popular method of evaluating the effectiveness
of a given design was conducting surveys among the study
participants. Additional evaluation metrics focused on quantitative
performance metrics on an evaluation task and subjective
experience (see Table 2). Here we give general
40 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022