IEEE Robotics & Automation Magazine - September 2015 - 34

physics, are many: multilevel causality,
the large number of preconditions and
laws involved, and probabilistic relations
between causes and effects. Robotics has
many problems in common with biology and medicine. We have comparison
and evaluation criteria for cars and
many other machines and appliances;
we are just starting to develop those for
robotics and intelligent systems. The articles in this issue show that it is possible,
and we already have some promising
proposals. After
several years of
This situation is bad
discussions and
attempts presentfor science, as it
ed in a long series
becomes difficult to
of workshops
and elsewhere, a
objectively evaluate
new kind of repthe state of the art
licable paper in
robotics has bein a given field.
come mature.
State of the Art?
There has been a growing awareness
about these issues in the community.
Yet, it is still very difficult to find examples of replicable papers in robotics and
automation. It is now possible to attach
supplemental materials to papers in the
most important journals of the field. Increasingly, authors share data sets and
code, in particular, in the simultaneous
localization and mapping (SLAM) community, and shared data sets and libraries, like Peter Corke's MATLAB
libraries, are made available. But despite
the progress in defining replication protocols, we are, in this respect, at the very
beginning. Years ago, Amigoni et al. [8]
showed that not a single paper among
the top cited ones in SLAM and navigation met all the basic criteria listed in
the GEM guidelines. We may have
clearly improved since then, but probably not enough.
Competitions have also matured in
the direction of becoming experiments
on the most elusive intelligent behaviors. You will not find the real state of
the art here, either in this editorial or in
this issue, as far as replication of results
is concerned. The reason is straightforward: this issue is the first example of a
publication including a list of replicable
34

IEEE ROBOTICS & AUTOMATION MAGAZINE

research results. To a certain extent, the
state of the art coincides with this issue
of IEEE Robotics and Automation Magazine (RAM).
It is also interesting to note that, in
more established areas of research with
more mature experimental methodology, like clinical research, there have recently been serious concerns about the
replicability of published research and
the consequent negative impact on research and even new drug development
and health care [3]-[5]. The idea that
the publishing process should evolve is
widespread. Published research reporting should provide enough information
to allow the replication of the results.
The web, and the easier distribution of
information that the web makes possible, might be part of the solution.
On the one hand, this new possibility was identified several decades ago
[6]. On the other hand, the practice of
sharing research is already evolving, as
shown by the success of preprint epublishing platforms like arxiv (www.
arxiv.org) or some recent experiments
of open review on the web (see http://
openreview.informatik.uni-freiburg.
de) as well as by this special issue.
Contribution of this Issue
After many discussions and attempts, a
new kind of paper seems to be necessary.
This new kind of paper should include
the following:
● description-a journal paper with
text, figures, and multimedia, according to GEM guidelines (or similar)
● data sets-similar to the option provided by various journals and magazines, included this one
● code identifiers-complete code identifiers and/or downloadable code
(executable files may be enough)
● HW identifiers-HW description or
HW identifier (if it is identifiable).
This special issue of RAM is the very
first example of a collection of replicable
robotics reports covering a remarkably
wide area of diverse robotics subfields.
The articles in this issue provide a living
example of the viability of replicable research in robotics. They span a wide
and diverse set of areas of research in
robotics, thus countering the idea that

September 2015

this field is too diversified to allow a rigorous shared methodology.
The articles report replicable experiments, benchmarking methods, and a
couple of exemplary surveys on competitions ("Humanoid Robots in Soccer," by
Reinhard Gerndt, Daniel Seifert, Jacky
Baltes, Soroush Sadeghnejad, and Sven
Behnke) and on the new and important
field of soft robotics ("Deformation in
Soft-Matter Robotics," by Liyu Wang and
Fumiya Iida). We have a very interesting
article about how competitions can be
given a rigorous scientific meaning in the
("Competitions for Benchmarking," by
Francesco Amigoni, Emanuele Bastianelli,
Jakob Berghofer, Andrea Bonarini, Giulio
Fontana, Nico Hochgeschwender, Luca
Iocchi, Gerhard K. Kraetzschmar, Pedro
Lima, Matteo Matteucci, Pedro Miraldo,
Daniele Nardi, and Viola Schiaffonati).
The set of replicable research examples
covers wearable systems ("Wearable Inertial Sensors," Barbara Bruno, Fulvio Mastrogiovanni, and Antonio Sgorbissa) and
manipulation ("Benchmarking in Manipulation Research," by Berk Calli, Aaron
Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar). We have three papers on different
aspects of marine robotics ("Tracking
Divers," by Nikola Mišković, Đula Nađ,
and Ivor Rendulić; "Exploring 3-D Reconstruction Techniques," by Javier Pérez,
Jorge Sales, Antonio Peñalver, David Fornas, José Javier Fernández, Juan Carlos
García Sánchez, Pedro J. Sanz, Raúl
Marín, and Mario Prats; and "Testing the
Waters," by Andrea Sorbara, Andrea Ranieri, Eleonora Saggini, Enrica Zereik,
Marco Bibuli, Gabriele Bruzzone, Eva
Riccomagno, and Massimo Caccia). And
then we cover motion planning ("Benchmarking Motion Planning Algorithms,"
by Mark Moll, loan A. Şucan, and Lydia E.
Kavraki), bipedal locomotion ("Benchmarking Bipedal Locomotion," by Diego
Torricelli, Jose Gonzalez, Jan Veneman,
Katja Mombaur, Nikos Tsagarakis, Antonio J. Del-Ama, Angel Gil-Agudo, Juan C.
Moreno, and Jose L. Pons), and last but
not least the requirements for replicable
simulation experiments ("RoboCup Simulation Leagues," by David M. Budden,
Peter Wang, Oliver Obst, and Mikhail
Prokopenko).

http://www.arxiv.org http://http://

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015