IEEE Robotics & Automation Magazine - September 2015 - 35

You should read the articles from
various standpoints: the novelty of the
content, the significance and viability of
the proposed benchmarks, the approaches that the authors have chosen to
allow the replication of their results. The
first question to ask is: are these results
reproducible? You will notice that the
articles have different focuses and that
the approaches are different. What are
the strengths and weaknesses of the
various approaches? Some authors, like
Perez et al., seem more focused on the
definition of the benchmarking criteria,
others, like Moll et al. or Sorbara et al. on
the replicability of the benchmarks.
Some, like Bruno et al., rely on thirdparty repositories like github or sourceforge, some have designed a dedicated
website. Some use an XML-based description, some do not. Are the experiment statistics always managed in the
best way? How should the statistical significance of the experiments be evaluated and the related metrics reproduced?
Have a look at Figure 1 in Moll et al.; to
be able to replicate the experiments, we
will need to structure systems like that.
What is the best way to implement
them? You may wish to compare with
Figure 2 in Sorbara et al. (for example).
This collection of very interesting
articles inspires a long list of thoughtprovoking questions and provides many
possible solutions and insights.
Of course, this is just a starting point.
Hopefully, the practical replication of
the results by the community will show

the best ways to provide information to
make the results of robotics papers
reproducible.
Road Ahead
We will need to foster the proper attitudes toward replication of results in the
community. We should not think that
scientific publishing could not further
evolve. Replicable papers can be a valuable addition to the current scientific
publishing landscape. In this new context, the initial severe peer-review preceding the publication of papers will be
just a prerequisite for the real peer-review based on the active reproduction of
the published results by the community
at large. This will also make easier the
understanding of the still open scientific
problems related to intelligent, animallike, and cognitive behaviors.
Another thing to consider for the
future is that the authors of the articles
in this issue, while usually providing
the information necessary through
their own websites, also had to upload
the data needed for replication to the
magazine website as attachments to this
article. We think that, in the future, we
will need a more structured approach;
in this sense, the website structures for
this issue will also contribute to the definition of a new publishing set of conventions to present replicable papers,
not as just attachments. This is what is
available now, and it is useful to have a
single self-contained entry for all the
articles in the special issue.

We would like to see the results of
many of the articles here reproduced as
they are in other articles commenting on
these issues and suggesting improvements. Although many challenges are still
ahead, we believe we are heading in the
right direction: back to the basics of the
scientific method.
References
[1] F. Bonsignorio, J. Hallam, and A. P. del Pobil,
Eds. (2008). GEM Guidelines. Euron GEM Sig Report. [Online]. Available: http://www.heronrobots.
com/EuronGEMSig/
[2] F. Bonsignorio, A. P. del Pobil, and E. Messina,
"Fostering progress in perfomance evaluation and
benchmarking of robotic and automation systems
[TC spotlight]," IEEE Robot. Automat. Mag., vol.
21, no. 1, pp. 22-25, 2015.
[3] Challenges in Irreproducible Research. Nature
Special. [Online]. Available: http://www.nature.
com/nature/focus/reproducibility/
[4] How Science Goes Wrong: Scientific Research
has Changed the World. Now it needs to change itself, The Economist, 2013.
[5] Trouble at the Lab: Scientists Like to Think of
Science as Self-Correcting. To an alarming degree,
it is not, The Economist, 2013.
[6] J. Claerbout, "Electronic documents give reproducible research a new meaning," in Proc. 62nd
Annu. Int. Meeting: Society Exploration Geophysics, 1992, vol. 92, pp. 601-604.
[7] F. Bonsignorio, J. Hallam, and A. P. del Pobil,
"Defining the requisites of a replicable robotics experiment," in Proc. RSS Workshop Good Experimental Methodologies Robotics, 2009.
[8] F. Amigoni, M. Reggiani, and V. Schiaffonati,
"An insightful comparison between experiments
in mobile robotics and in science," Auton. Robot.,
vol. 27, pp. 313-325, 2009.

Are You Moving?
Update your contact information so you don't
miss an issue of this magazine!

IMAGE LICENSED BY INGRAM PUBLISHING

Change your address
E-MAIL: address-change@ieee.org
PHONE: +1 800 678 4333 in the United States
or +1 732 981 0060 outside
the United States
If you require additional assistance regarding your IEEE mailings,
visit the IEEE Support Center at supportcenter.ieee.org.
IEEE publication labels are printed six to eight weeks in advance
of the shipment date, so please allow sufficient time for your publications
to arrive at your new address.

September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

http://www.heronrobots http://www.nature http://supportcenter.ieee.org

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015