IEEE Robotics & Automation Magazine - September 2015 - 33

or, better, scenario-based evaluation
procedures, have been recognized as a
component of the recipe for the benchmarking of results, particularly when intelligent behaviors are involved. The
extent to which competitions can be regarded as scientific experiments, and
which ones, is still a matter of discussion. An article in this issue, "Competitions for Benchmarking," by Francesco
Amigoni, Emanuele Bastianelli, Jakob
Berghofer, Andrea Bonarini, Giulio Fontana, Nico Hochgeschwender, Luca Iocchi, Gerhard K. Kraetzschmar, Pedro
Lima, Matteo Matteucci, Pedro Miraldo,
Daniele Nardi, and Viola Schiaffonati,
may provide some hints.
Methodological, Practical,
and Epistemological Issues
Although the number of robotics papers
published in journals and conferences is
constantly growing, the possibility of reproducing results is left to the good will of
some authors. The number and nature of
envisioned applications and proposed
methods are vast and also steadily increasing. As a consequence, some members of the community believe that the
comparison of results would not be practically possible. A remarkably varied set
of robotic applications is approached by a
significantly disparate set of methods,
sometimes based on notably different
principles, with different hardware
(HW)/software (SW) architectures in different environments. On the one hand,
the explosive growth of research results
shows that the community is becoming
larger and increasingly active; on the
other hand, it raises some serious problems when you have to objectively evaluate the actual relevance of the results and
the actual state of the art in a given field.
As previously stated, the difficulty of
reproducing results-let alone comparing different methods and solutions-
slows down the industrial take-up of
new solutions. Basic research is also
hindered, since it is very difficult for a
research group to build on the results of
another one, leading to a very limited
cross-exploitation of results between
different groups, and a general prevalence of exploration over exploitation.
Many new solutions are proposed, but

the community often does not go deep
into the analysis of most of them.
The EURON GEM guidelines [1]
are essentially an adaptation to the robotics and automation domain of the
general methodology applied in science
and engineering that was pioneered by
Galileo and Boyle. Today, as discussed
in [2], only a limited subset of published
results follow those methods and usually not completely. Of course, not every
paper should follow a rigorous experimental protocol: position papers, concept papers describing upcoming
research, papers concerning algorithms,
or survey papers do not need to comply
with a rigorous and epistemologically
sound experimental methodology. Still,
many papers that claim to have solved a
problem (say, autonomous driving)
based on simulations or field experiments should comply. Robotics, artificial intelligence, and automation are not
pure mathematics. The proposed solutions need to be able to work in the set
of environments and for the set of tasks
for which they have been studied. There
are scientific aspects in robotics, for example, related to the unbundling of the
brain-body nexus in humans and animals, but even when we are closer to
pure engineering applications, experiL1, L2, L3, ..., Ln

Covering Laws

C1, C2, ..., Cn

Explanans
Initial Conditions

Explanandum

Figure 1. The Hempel-Oppenheim model
of scientific knowledge. In the conceptual
schema represented in this figure, which
summarizes the Hempel-Oppenheim
model of scientific knowledge, all the logical
enunciates have a probabilistic truth value.
We need a precise and complete list of laws
invoked for the explanation, a precise and
complete list of initial conditions (system
HW/SW architectures, environments, tasks),
a precise definition of what is explained
or proved. In addition, we must accept the
fact that our theoretical claims, enunciate,
have to be of probabilistic nature, since
we operate in open-ended stochastic
environments. (Figure adapted from G.
Boniolo, "A Contextualized Approach to
Biological Explanation," Philosophy, vol. 80,
pp. 219-247, 2005).

mental proofs of the effectiveness of the
proposed solutions are needed.
We should at least be able to
● validate the results by replicating
them
● compare the results in terms of the
chosen performance criteria.
This holds true for both purely scientific
issues and real-world applications. The
fact that robotics research deals with very
diversified problems should not be seen
as a serious obstacle. Indeed, medicine
and life science,
for inst ance,
where the comWe should not be so
plexity and varisurprised by the fact
ety of the studied
objects are not
that we are struggling
smaller than in
to define valid and
robotics, have developed rigorous
shared benchmarking
e x p e r i m e nt a l
procedures for
protocols. We
should take inintelligent robots.
spiration from
them. An epistemological model of biological science
was proposed by Hempel and Oppenheim; see Figure 1.
We can expect that having replicable
and measurable results will affect the
content of the results, not just their reporting. We should not be so surprised
by the fact that we are struggling to define valid and shared benchmarking
procedures for intelligent robots. Their
development uncovers a lot of practical,
publishing, and also epistemological issues. A more detailed discussion of this
topic can be found in [7] and will be the
main topic of a future publication. Besides the so-far unsatisfactory, in this respect, experimental and reporting
practice, an important reason could be
the limited scientific understanding of
intelligence and cognition in natural and
artificial systems. The practical issues
span from modeling, to statistical significance assessment, to the mechatronic
design and construction of specific test
equipment, and to the actual replication
procedures, the experimental protocols,
and the necessity to provide the data,
time series, and HW/SW description.
The epistemological issues, with respect
to paradigm examples of science, like

September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015