IEEE Robotics & Automation Magazine - September 2015 - 56

speech understanding). A TBM deals with complete robot
systems, implying that a large set of interacting robot subsystems (like navigation, perception, and manipulation) are examined together at the same time. FBMs, on the other hand,
focus on the performance of single subsystems, defining a
precise setup in which a single robot functionality can be
evaluated. This evaluation is performed according to wellspecified quantitative measures and criteria, which depend
on the functionality being tested. The scoring of TBMs and
FBMs is then used to determine rankings of teams and to
award prizes at the competitions.
The RoCKIn approach builds on the efforts of the
RoboCup community to identify which functionalities are
stable and solved or unsolved, at least in the context of the selected competitions.
However, the evaluation
The replicability can
of these functionalities in
the RoboCup is mixed
be specified into
with the evaluation of the
tasks, and separating the
reproducibility and
two is difficult. "In fact,
teams obtained good rerepeatability.
sults in navigation, mapping, person tracking,
and speech recognition
(with the average above 50%, except for navigation). Notice
that the reason for a low-percentage score in navigation is
not related to inabilities of the teams, but it is because part of
the navigation score is only available after some other task
was achieved [13]." The RoCKIn approach avoids this problem by limiting the influence of all other subsystems when
evaluating a robot functionality in a FBM. For example, for
testing object perception, robots are put in place before starting the test. More generally, we mitigate the difficulty of separating subsystems under investigation from their
environments, which also include other robot subsystems

Table 2. The teams participating in
the 2014 RoCKIn competition.
RoCKIn@Home Teams
b-it-bots@Home, Bonn-Rhein-Sieg University of Applied
Sciences, Germany
BARC, University of Birmingham, United Kingdom
Homer@UniKoblenz, University of Koblenz-Landau, Germany
Pumas@Home, Universidad Nacional Autonoma
de Mexico, Mexico
SocRob@Home, Universidade de Lisboa, Portugal
UrsusTeam, University of Extremadura, Spain
Watermelon Project, University of Leon, Spain
RoCKIn@Work Teams
b-it-bots@Work, Bonn-Rhein-Sieg University of Applied
Sciences, Germany
IASLab@Work, University of Padua, Italy
SPQR@Work, Sapienza University of Rome, Italy

IEEE ROBOTICS & AUTOMATION MAGAZINE

September 2015

that are not being evaluated, by carefully designing FBMs
that minimally involve these last subsystems.
The TBMs and FBMs are evaluated differently. The TBMs
measure the achievement of goals, which is a yes or no answer to specific questions (e.g., "Does the robot understand
Annie's command(s)? Does it correctly identify the requested
object?"). The FBMs measure robot performance, which is a
number resulting from the measures used for scoring and
ranking, such as effectiveness (e.g., precision and recall) and
efficiency (time, resources used, and so on), as further discussed in the next section. This division resembles the recently proposed evaluation of artificial intelligence systems
[12], which is based on task-oriented and ability-oriented
evaluations. In both the cases, the RoCKIn approach tries to
avoid subjective evaluation to improve reproducibility. Indeed, attention has been dedicated to requirements for
benchmarking and scoring runs as autonomously as possible
(i.e., without continuous human intervention), specifically by
using automated computing systems called RefBoxes (also
called Central Factory Hub in RoCKIn@Work). In this respect, the more automated scoring approach of RoCKIn contrasts, for instance, with that of the RoboCup Rescue Robot
League [17], which is heavily based on human judges.
In addition to ranking teams in the competitions, the approach of RoCKIn promises to be a good way to understand
robot systems because it enables researchers to study the impact of functionality performance on task performance.
Moreover, it forces teams to develop means of continuously
monitoring the performance of their robot systems because
they have to provide regular feedback to the RefBoxes and
store data for benchmarking.
The First RoCKIn Competition
The first RoCKIn competition (http://rockinrobotchallenge.
eu/rockin2014.php) was held in Toulouse, France, 26-30
November 2014, and was the first opportunity to test the
practical application of the approach outlined in the previous
section and to prepare for the final RoCKIn competition
(http://rockinrobotchallenge.eu/rockin2015.php) to be held at
the end of 2015 in Lisbon, Portugal. The teams participating
in the 2014 event are listed in Table 2.
Setup for the Competition
The detailed specifications of the RoCKIn@Home and
RoCKIn@Work test beds are reported in the corresponding
rule books, which are available at http://rockinrobotchallenge.eu/publications.php under "deliverables and reports"
and allow for their precise reproduction at other sites than
those of the competitions. The layouts and the sizes of the
RoCKIn@Home and RoCKIn@Work arenas (Figures 1 and
2) are fully specified, together with the materials of walls and
the precise definition of objects present in the environments
(at a level of detail that include furniture and floristic objects
for RoCKIn@Home). The robots must conform to certain
size, weight, and safety restrictions and can be wirelessly networked with other devices. Apart from this, teams are free to

http://rockinrobotchallenge http://www.rockinrobotchallenge.eu/rockin2015.php http://rockinrobotchal http://www.lenge.eu/publications.php

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015