IEEE Robotics & Automation Magazine - September 2015 - 191

Q: This special issue on reproducibility and measurability of robotics
research demonstrated a high interest
from the community of the IEEE
Robotics and Automation Society
(RAS). What opportunities do you
foresee for linking this interest to
future initiatives in this area which
could be launched at the IEEE level?
GS: Developing the necessary infrastructure for supporting RR as well as the
best practices associated with it (e.g., in
terms of the review process of the data,
code, and algorithms associated with the
paper) will require substantial work and
support from many different IEEE communities. The RAS has already made significant steps in these directions and its
experience will be truly precious for the
entire organization.
Q: Do you consider research in the

robotics and automation domains a
key area for applying the principles,
methods, and tools for aiming at RR?
GS: Absolutely. Robotics and automation is one of the best areas to apply
and test any best practices that the IEEE
will develop in terms of RR. In fact, the
robotics and automation domains rely

on mathematical science as well as computational and experimental ones, so
that these experiences pertain to all
kinds of reproducibility mentioned in
the answer to the second question.

Q: Do you consider these topics
important for the training of a
new generation of researchers in
engineering?
GS: I consider them fundamental.
Adopting RR will, in fact, truly change
the culture and will require substantial
additional effort from the authors publishing with the IEEE. This is, of course,
a process that cannot be enforced but
only reinforced. We need, therefore, to
educate the community, especially the
young professionals, to comprehend and
embrace the benefits that RR can bring
from all different perspectives: authors
(visibility increase), users (enhancement
in the exploitability of results, increase in
capability of recognizing fundamental
results) and humanity as a whole
(increased rate of innovation).
References

[2] J. B. Buckheit and D. L. Donoho, "WaveLab
and reproducible research," Dept. of Statistics,
Stanford Univ., Stanford, CA, Tech. Rep. 474,
1995.
[3] H. W. Turnbull, Ed., The Correspondence of
Isaac Newton: 1661-1675, vol. 1. London, U.K.:
Royal Society at Univ. Press, 1959, p. 416.
[4] Special online collection: Hwang et al. controversy-Committee report, response, and background.
Science. [Online]. Available: http://www.sciencemag.org/site/feature/misc/webfeat/hwang2005/
[5] C. G. Begley and L. M. Ellis. (2012). Drug
development: Raise standards for preclinical
cancer research. Nature. [Online]. 483(7391),
pp. 531-533. Available: http://www.nature.com/
nature/journal/v483/n7391/full/483531a.html
[6] J. P. A. Ioannidis et al. (2009). Repeatability of
published microarray gene expression analyses.
Nature Genet. [Online]. 41(2), pp. 149-155.
Available: http://www.nature.com/ng/journal/
v41/n2/full/ng.295.html
[7] How science goes wrong. (2013, 19 Oct.). The
Economist. [Online]. Available: http://www.economist.com/printedition/2013-10-19
[8] H. A. Piwowar, R. S. Day, and D. B. Fridsma,
"Sharing detailed research data is associated with
increased citation rate," PLoS One, vol. 2, no. 3,
e308, 2007, DOI: 10.1371/journal.pone.0000308.

[1] Wikipedia. Reproducibility. [Online]. Available: http://en.wikipedia.org/wiki/reproducibility

From the Field (continued from p. 188)
Importantly, the system also allows
researchers to create their own specialized, operational tasks to run in natural
settings. We claim that this method can
be applied to all kinds of applications.
An opportunity to try this out arose in
July 2014 at the Arkaroola Mars Robot
Challenge organized by the Mars Society Australia. Four student teams
brought six field robots to a test site in
Arkaroola, a remote desert station in
South Australia. The machines embodied the students' design concepts for
assistant robots for astronauts performing tasks on the Martian surface (Fig-

ure 1). A selection of six standard
DHC-NIST-ASTM benchmarks,
together with three operational tests
specific to surface operations in harsh
Mars-like terrain, was conducted over
12 days. For example, we had the
robots search a gullied slope for a hidden target object, which had to be photographed, collected, and returned to
the operator.
The test details and results will be
formally presented in September at Towards Autonomous Robotic Systems
2015 in Liverpool, United Kingdom,
but in brief, we found that most, but

not all, tests worked well, provided one
practices the procedures and allows
enough time (Figure 2). We were not
only able to gather a good deal of standard performance data, but we were
able to use it later to make real design
improvements to two of the robots.
Our test program could accommodate
unmanned aerial vehicles as well as
ground machines: one participant was
able to score highly on many tasks
using a small quadrotor, suggesting
very high utility of a (suitably modified) drone for future Mars explorers.

September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

191

http://www.sci http://www.encemag.org/site/feature/misc/webfeat/hwang2005/ http://www.nature.com/ http://www.nature.com/ng/journal/ http://www.econ http://www.omist.com/printedition/2013-10-19 http://en.wikipedia.org/wiki/reproducibility

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015