IEEE Robotics & Automation Magazine - June 2020 - 165

Results: PANGAEA-X Field Test
Our field experiment was conducted in Lanzarote, Spain. The
unique landscape of this volcanic island is one of the closest to
a lunar landscape one can find on Earth, and the ESA uses it
for geology training for astronauts.
Our sample was small (five participants) due to logistic
and weather constraints: setting up the UAV fleet took
approximately 30 min per run, each trial lasted about an hour,
and UAVs have weather criteria for safe operation (visibility,
light, precipitation, and wind speed). We first performed a
series of experiments on our own team members (beta testers); then we conducted a core set of experiments on two
members of the ESA crew and one journalist.
Informed consent was obtained. The eye tracker was calibrated [43]; then, the operators explored an area for hidden
ground features by guiding the UAVs using each of the two
control modes, in separate missions, while listening and
responding to intermittent radio chatter. After each of the two
missions, the participants' feedback on their user experience
was collected.
Exploration Task Performance
The aim of each mission was to search for hidden ground features, a task best accomplished by widely covering the search

Waypoints

Self-Deployment

150
100
y (m)

Our experiment would not be realistic if we ignored all of
the other tasks that human operators would need to perform
in parallel for planetary exploration. We created fictional team
radio chatter, which is played over earphones [Figure 3(b)].
The audio files contain contextual information, such as "John
is going out on EVA; keep an eye out," and occasional mission-specific questions prefaced with the call sign "Operator,"
such as "How far is robot one from the initial point?" The
operator is instructed to quickly acknowledge the communication with a button press on the control screen and respond
verbally. In this initial sample, the supplementary task serves
to ensure that the speech is attended to, thus increasing workload; in larger sample sizes, it would be possible to use missed
versus correct responses as an additional performance measure for statistically comparing conditions.
Cognitive load can be measured through subjective selfassessment metrics, such as questionnaires, and objective
metrics, such as body motion, heart-rate variability, and measures of pupil dilation over time derived from pupillometry
[37], [38]. Pupillometry has recently gained popularity in
applied psychology as a reliable proxy for cognitive load [39].
For example, pupil dilation was used to study the effects of
audiovisual interference on workload in piloting tasks [40].
We used both types of measurements: 1) a questionnaire,
which included questions inspired by a survey originally
designed to evaluate the perceived usability and acceptance of
assistive devices [41] augmented with task-load-oriented
questions from the NASA Task-Load Index, and 2) pupil dilation and variability [Figure 3(a)]. To provide equivalent psychological distance between the scores [42], all answers were
on a seven-point Likert-like scale (range: 0-6).

50
0
−50

0 50 100
x (m)
(a)

−50

0 50 100
x (m)
(b)

Figure 6. The top views of five UAV trajectories (different
colors) comparing the two control modes from a representative
participant. (a) The more manual waypoint condition shows
duplicated paths for different UAVs. (b) The more autonomous
self-deployment condition shows better area coverage. UAVs
were twice pushed beyond the geofence by strong winds in the
self-deployment condition (b).

area. A representative example of the UAVs' trajectories for
the two control modes is shown in Figure 6. The area covered
in the self-deployment mode is larger than in the waypoint
mode, whereas, in the waypoint mode, there are duplicate trajectories, indicating lower overall search efficiency. This outcome is expected since the self-deployment algorithm aims at
spreading the robots over the area of operator-defined interest. In the waypoint mode, human operators must place the
waypoints to sweep the area while mentally keeping track of
the area that each robot has already explored.
On average, for both control modes, the participants discovered most (three of the five total) of the hidden ground features;
testing a larger sample may reveal differences. Anecdotally,
most participants expressed confusion about which areas had
already been explored in the waypoint condition, and they also
relied on their memory to assign new goals to the UAVs
(instead of clicking on a UAV to get its ID), often sending an
unintended UAV toward a new goal. Similar observations apply
to the trajectories obtained by the other participants.
Perception of Usability and Workload Over Levels
of Swarm Autonomy
Figure 7 presents the average results of the main survey elements for both control modes. For both, the interface was
considered similarly easy to learn (>4.6), easy to use (>4.4),
intuitive (>4.4), and effective (=4.4) without being cumbersome (<1.5). We attribute these promising initial results to the
minimalist and clutter-free design of our mission planner [see
Figure 4(b)] and to the good will of the participants. The
results also show that neither task was considered highly
demanding (<3.0) or hard to complete (<3.0).
Figure 7 shows that individual waypoint control gives the
operator more confidence (less insecurity) about the outcome
of his or her actions. Results also show that the self-deployment
mode was less intuitive, less efficient, and harder to learn. We
can also observe that this control mode was slightly more mentally demanding. We believe these perceptions are related to the
JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

165
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
