IEEE Robotics & Automation Magazine - June 2018 - 66

Hypothesis 1.2: The effectiveness of a human-robot
team in accomplishing a collaborative task will be higher when the human subjects receive visual feedback as
they perform and complete tasks rather than having no
feedback. Communicating information and instructions
visually and in the right place at the right time is faster and
more intuitive and improves overall task performance. In
contrast, instructions displayed on a mobile device or in
the form of printed texts might cause ambiguities to arise
in a real-time task situation. We defined efficiency as the
time taken for the human subjects to complete the task and
effectiveness as the accuracy percentage of task completion.
● Hypothesis 2: The time taken by each human subject to
understand a specific task will be constant when the
instructions are in the form of just-in-time visual cues. In
contrast, there will be
high variation in understanding times between
The total task completion
human subjects when the
instructions are printed
time was found to be
on a paper or displayed on
a mobile device. We anticlower in the projection
ipate that different human
subjects need more or less
case as compared with the
the same amount of time
to understand clear and
measured values from the
concise information in
augmented visual form.
printed mode and mobile
We also expect to see
large variations in taskdisplay mode.
understanding times
between subjects relying
on the printed version. To test this hypothesis, we measured the time taken for each subject to read or interpret a subtask in each task condition and then compared
the measurements.
● Hypothesis 3: Subjects will be more satisfied collaborating
with the robot in the projection mode than in the other
two modes. Additionally, explicit visual feedback will instill
a positive attitude in human subjects. In contrast, subjects
will feel negative or neutral when they receive no explicit
feedback from the system or robot. It is important to provide the human subjects with feedback regarding the
robot's intention and the subject's action. This, in turn,
ensures that the human collaborator will feel comfortable
and satisfied working with the robot. To obtain the subjective measurements, human participants completed a posttest questionnaire consisting of a series of Likert scale and
free-response questions.
●

Experimental Methods
We asked subjects to collaborate with a robot to carry out a
well-specified assembly task in a simulated manufacturing
environment. The joint assembly task involved a human subject and a stationary manipulator with 6 DoF (a UR5 robot)
performing a total of 12 manipulation steps on a car door. The
66

IEEE ROBOTICS & AUTOMATION MAGAZINE

june 2018

assembly process required removing new components and
tools from a set of tool boxes, connecting components in a specific order, and finally attaching them at different locations on
the door. The car door was placed on a caster and could be
moved to different locations. All of the experiments were
reviewed and approved by the Institutional Review Board at
Arizona State University, Tempe. A video demonstrating our
experiment can be found in [31].
Experimental Procedure
First, the participants were briefed on the experiment and the
assembly task scenario. Subjects were informed that 1) they
had to collaborate with the robot in completing a procedural
task consisting of 12 subtasks that needed to be finished successfully in sequence and that 2) failing to complete one subtask would result in failing one or more subsequent subtasks.
Nine of the 12 subtasks were assigned to the participants, while
the rest were given to the robot. The order of the subtasks was
partially randomized in all three conditions (printed, mobile
display, and projection mode). Each subject carried out a total
of three task trials under each condition. All of the participants
were required to read and sign a consent form before beginning the experiment.
Experimental Task
The goal of the experimental task was to assist the robot in
assembling a car door in a simulated manufacturing environment. The task involved carrying out a set of sequential subtasks x = {x 1, x 2, f, x 12} in a specified order. A subtask x i
could be any one of the following:
● pick an assembly part (an interchangeable part) or tool
● place an assembly part or tool
● move the car door to a specified location inside the
workspace
● align the car door with a specified reference point
● join assembly parts together
● screw assembly parts onto the car door.
The instructions to execute the subtasks were framed as
sequential steps and provided to the participants as printed,
mobile display, or projected instructions, depending on the
test condition. The instructions also specified whether the
subtask was to be completed by the human or the robot.
Measurement Instruments
The entire experiment was videotaped for post hoc analysis.
The efficiency and effectiveness were evaluated objectively
by measuring the completion time and accuracy of each
subtask. Subtask completion time, for both human and
robot, was measured by recording the difference in time
between the start and end of the subtask. For a human subject, the subtask completion time was expressed as the total
time spent on understanding the instructions and then executing them.
The percentage of task completion (the fraction of successfully completed subtasks) was used as a measure to evaluate
the effectiveness of the collaborative task. Additionally, the

IEEE Robotics & Automation Magazine - June 2018

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2018