IEEE Robotics & Automation Magazine - September 2015 - 32

from the guest editors

Toward Replicable and Measurable
Robotics Research
By Fabio Bonsignorio and Angel P. del Pobil

he famous experiment by Galileo-one of the founders of
modern science-in Pisa's
Cathedral in 1582, was one of
the very first examples of a scientific
experiment validating a scientific
result: the discovery of the pendulum
law. Galileo measured the variations of
the oscillation period of a lamp in the
dome by his own heart rate. From
those times, experiment replication
and experiment replication and reproducibility of results are at the cornerstone of the scientific method. Yet in
robotics, artificial intelligence, and automation, the reproduction of results
from conference and journal papers, as
they are today, is quite often very difficult, if not impossible. This situation is
bad for science, as it becomes difficult
to objectively evaluate the state of the
art in a given field, and also it becomes
problematic to build on other people's
work, thus undermining one of the
basic foundations of scientific progress.
Moreover, it is detrimental to the industrial exploitation of results, for
which we need to compare the effectiveness and efficiency of different
methods proposed to solve the same
technical or scientific problem, for example, from the computational and energy-consumption standpoints. This
difficulty in reproducing results, however, makes this comparison usually very
cumbersome and without trustable outcomes. This situation hampers and
slows down the industry take-up of reDigital Object Identifier 10.1109/MRA.2015.2452073
Date of publication: 11 September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

search results, and there are many more
than those already exploited that are
likely to benefit our daily lives.
The community has been aware of
this issue for a long time. In 2007, we,
with John Hallam, created the European Robotics Research Network
(EURON) Good Experimental Methodology (GEM) and Benchmarking
Special Interest Group (SIG) within the
EURON Network of Excellence (NoE),
a NoE is a networking-oriented European-funded project. In 2006, one of
us, Angel P. del Pobil, organized a
workshop on benchmarking at the
IEEE/Robotics Society of Japan International Conference on Intelligent Robots and Systems (IROS) in Beijing,
China, as an activity of the EURON
NoE work package devoted to benchmarking, and the first website on survey and inventory of current efforts in
comparative robotics research was established (http://www.robot.uji.es/
EURON/en/index.htm).
The GEM guidelines [1] were one of
the major outputs of the SIG's early activities. Although, initially, the guidelines
were focused on more careful reviews,
mainly thanks to one of us (Fabio Bonsignorio), the real problem became
clear: the core issue is the reproducibility/replicability of experimental results.
In 2009, at the International Conference
on Robotics and Automation (ICRA) in
Kobe, Japan, the IEEE Robotics and Automation Society (RAS) Technical Committee on Performance Evaluation and
Benchmarking of Robotic and Automation Systems was founded, with similar
objectives. In parallel, the Performance

September 2015

Metrics for Intelligent Systems conference series focuses on performance
measurement challenges arising from
the application of robotics and automation technologies to practical problems
in the commercial, industrial, homeland
security, and military domains. More information can be found at http://www.
nist.gov/el/isd/permis2012.cfm.
There has been a long series of workshops at various conferences, such as
IROS, ICRA, and the Robotics Science
and Systems Conference, in which more
than 200 people have participated so far.
We mostly organized them with Elena
Messina and John Hallam, but there
have also been some organized by others, and there have been a number of
competitions and publications aiming at
finding a way out of a situation that is
considered by many as unsatisfying (see
http://www.ieee-ras.org/performanceevaluation and http://www.heronrobots.
com/EuronGEMSig/).
When EURON joined euRobotics
Association Internationale Sans But Lucratif (AISBL), the private part of the
European Public-Private Partnership on
Robotics, the activities of the former
EURON GEM SIG became part of the
Topic Group on Evaluation and Assessment of Research Results, also known as
Benchmarking and Competitions. A
solid example of benchmarking methodology is proposed in "Benchmarking
in Manipulation Research" by Berk
Calli, Aaron Walsman, Arjun Singh,
Siddahrta Srinivas, Peter Abbel, and
Aaron Dollar. There are experimental
setting where this approach is difficult to
implement. In those cases competitions,

http://www http://www.nist.gov/el/isd/permis2012.cfm http://www.ieee-ras.org/performance http://www.heronrobots http://www.robot.uji.es/

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015