IEEE Robotics & Automation Magazine - June 2018 - 68

the three task trials had different sets of subtasks, even
though only the order of the subtasks was randomized. To
eliminate order effects, the order of the modes (printed,
mobile display, and projected) was also randomized for different
The average task
groups of participants.

completion percentage is

Objective Findings

significantly higher in the

Efficiency
Hypothesis 1.1 states
that the efficiency of the
human-robot collaborain the printed and mobile
tive team will be higher
in the case of the prodisplay modes.
jected condition when
examined against the
printed and mobile display modes. The total time taken to
finish all of the subtasks was measured and compared for
the three conditions. The total task completion time was
found to be lower in the projection case as compared with
the measured values from the printed mode and mobile
display mode. Figure 7(a) illustrates the average task completion time for all three test conditions.
An analysis of variance, using the one-way ANOVA test,
showed statistically significant differences in total task completion times among the different task conditions, with
F (2, 42) = 8.07, p 1 0.01. The task completion time in
the projected condition ( n = 467.73, v = 135.22) was
lower than the time in the printed condition ( n = 678.60,
v = 165.60), t (14) = 8.02, p 1 0.00001; and the mobile
display condition (n = 606.53, v = 135.59), t (14) =
6.31, p 1 0.0001.
The statistically significant results reinforce our hypothesis that human-robot teams are more efficient with justin-time projected instructions than with printed or
displayed instructions.

projected condition than

Effectiveness
We assessed the effectiveness of the task in the three test conditions by considering the percentage and accuracy of task
completion in each test scenario. The percentage of task completion by the human-robot team was computed as the fraction of successfully completed subtasks out of all of the given
subtasks. We compared the three conditions using a one-way
ANOVA test and found statistical differences in the task completion percentage as a function of the mode of communication, with F (2, 42) = 7.26, p 1 0.01. It can be seen from
Figure 7(b) that the average task completion percentage is significantly higher in the projected condition than in the printed and mobile display modes.
As a measure of accuracy, we recorded the ground-truth
errors for subtasks involving the alignment of the car door
and objects in both task conditions. Our experiment included
four error-measurable subtasks-three instances of car door
alignment and one circular object alignment-that involved
measuring translation and rotation errors. Both translation
and rotation errors were comparably smaller in the projected
condition as compared to the printed and mobile display
modes. The one-way ANOVA test was used to analyze the
variance in the translation errors, showing that there is a statistically significant difference between the three conditions.
In comparison, a one-way ANOVA test on the rotation
errors revealed that all of the tasks, except car door alignment 3,
showed a significant difference between conditions, as illustrated in Figure 8. This is acceptable, because subtask 3 involved
rotating and aligning the car door parallel (0°) to the robot,
which is relatively easier to accomplish, even without feedback,
as compared to other subtasks that involved rotating the car
door to a specified angle.

Time (s)

Task Completion (%)

Task Understanding Time
In hypothesis 2, we postulated that the time taken by different subjects to understand a subtask will be constant if
the instructions are provided in augmented visual form. To
investigate this hypothesis, we measured the understanding times of the subjects in nine subtasks assigned to the participants,
and we analyzed the standard errors
of the means. Task understanding
800
100
time is defined as the time spent by
the participant in reading or looking
80
600
at instructions.
60
We observed that the standard
400
errors for all of the subtasks in the pro40
jected mode were significantly lower
200
20
than in the printed and mobile display
approaches, implying that most partic0
0
ipants took a similar amount of time to
Printed
Mobile Projection
Printed
Mobile Projection
Display
Display
understand a subtask. In contrast,
standard errors in the printed and
(a)
(b)
mobile display condition were comparatively higher, particularly for subFigure 7. The mean and standard error for (a) the task completion times and (b) the
percentages of task completion.
tasks 4, 8, and 9, as shown in Figure 9.

IEEE ROBOTICS & AUTOMATION MAGAZINE

june 2018

IEEE Robotics & Automation Magazine - June 2018

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2018