IEEE Robotics & Automation Magazine - September 2018 - 4

from the editor's Desk

On Reproducible Research
By Bram Vanderborght

uring the IEEE Panel of Editors
meeting held this past April in
Los Angeles, California, I was
invited, as editor-in-chief of
IEEE Robotics and Automation Magazine,
to participate in a panel discussing reproducible research under the lead of
John Baillieul, chair of the Strategic
Planning Committee. Reproducibility of
scientific research is imperative: it helps
researchers verify results and benchmark
them, and it allows others to build on
them, advancing the global body of
scientific knowledge.
We are not alone in this. Some even
speak of a "reproducibility crisis," in
which researchers have found that the
results of many scientific studies are difficult or impossible to replicate (an issue
that is especially prevalent in social psychology and medicine).
Within IEEE publications, reproducibility is still far from being ensured.
The panel was a continuation of the
IEEE's efforts to understand new and
emerging concepts of research curation
and how these are related to topics such
as research reproducibility. Without
question, this will over time become a
key requisite for articles appearing in
IEEE publications.
During the last IEEE International
Conference on Robotics and Automation (ICRA), held in Brisbane, Australia, our Society's efforts to ensure reproducibility were highly valued thanks to

Digital Object Identifier 10.1109/MRA.2018.2855338
Date of publication: 12 September 2018

IEEE ROBOTICS & AUTOMATION MAGAZINE

the leading role of our associate editor, Fabio Bonsignorio
and colleagues, who began
work on this topic ten years
ago and has since organized
several related workshops. Remember, we had a special issue in
September 2015 ("Replicable and Measurable Robotics Research") and in March
2017 ("Open-Source Movement"). But,
of course, the best is still to come, and I
was pleased to see that, for example,
Ken Goldberg during his keynote talk
at ICRA presented the advantages of
open benchmarks to advance the field
of robot grasping.
Therefore, in September 2017, we
launched the R-articles concept. In addition to publishing the description in the
journal paper text (written following the
"Good Experimental Methodology"
guidelines), the required data sets, the
complete code identifiers, and the hardware description of the simulation models, independent researchers are given
the opportunity to publish a short article
about the results replication in an R-article. The IEEE collaborates with CodeOcean, a cloud-based executable
research platform that provides researchers and developers an easy way to
share, discover, and run code published
in academic journals and conferences
and is linked with IEEE Xplore. The
paper "Caging Polygonal Objects Using
Equilateral Three-Finger Hands" by
Bunis et al. in IEEE Robotics and Automation Letters is the first related to
robotics that shared the code over
CodeOcean.

september 2018

Again, I offer a warm call
to submit articles with reproducible research. Together
with the authors, we need to
improve the procedure over
time, but we can only learn
from experience.
With full awareness that it requires
time and effort that may be in opposition to current publish-or-perish pressure, we need to increase efforts to
reward the positive behavior of authors
contributing to reproducible research.
One initiative suggests highlighting
paper with reproducible research content, perhaps even granting awards
dedicated to such papers. Discussions
are in progress with the Association for
Computing Machinery about the adoption of a common badging system for
such papers. A culture shift will be
needed. Moreover, along the lines of the
open access policy, funding agencies
may start implementing such practices.
With a stick? A carrot? It probably does
not matter, because we need to move
forward in the way we conduct our research to advance robotics for society
and the world economy.
This September issue includes all
regular articles (the three upcoming
issues will mostly be special issues). Pay
particular attention to the article on the
commercial sociable humanoid robot
Pepper from SoftBank Robotics. With
this article, we hope to receive more
submissions from companies describing their scientific results and innovations. Enjoy the issue!

IEEE Robotics & Automation Magazine - September 2018

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2018