IEEE Robotics & Automation Magazine - September 2017 - 180

formats. We obtained a large number of
high-quality submissions, ranging in
topic from manipulation to wearable systems, marine robotics to bipedal locomotion. The success of the special issue
showed that the community is ready for
this. We are now defining some rules and
have designed a full publishing process to
move from an initial pioneering effort to
a more organized process.
The RAM R-Article
We have decided to introduce a new
kind of article in the magazine, an article specifically designed to be reproducible, and we are calling it the R-article.
The authors will have to share
whatever they believe is necessary and
sufficient to reproduce their results
and confirm that in the cover letter. Of
course, since experimental outcomes
are produced in statistical terms, in
most cases it would be almost impossible to achieve exactly the same results,
i.e., the same inputs and the same
results. Moreover, the description part
of the report will at least have to comply with the criteria outlined, e.g., in
[1], and summarized in Figure 3.
This new kind of article should
include, at least,
● a description, i.e., an almost ordinary
journal article with text, figures, and
multimedia, complying at a minimum with the good experimental
methodology (GEM) guidelines
● data sets stored in CodeOcean, and
all the necessary information tests
that have been run; only in welljustified special cases will we accept
different sharing platforms; videos of
experiments should not be considered as data sets but as multimedia

one of the main bottlenecks for replication of robotics experiments.

Figure 2. The September 2015 RAM
special issue cover.

attachments if they are not actually
part of the data elaborated by some
algorithm in the robot software
● complete code identifiers and preferably downloadable code (executable files may be enough) on CodeOcean-the algorithms as they have
been coded and all necessary library
and middleware information; also in
this case, only in well-justified special cases will we accept different
sharing platforms
● a hardware (HW) description or HW
identifier (we need to know the physical details of the robots to be able to
repeat robotics experiments); by HW
description, we mean the full design
(at the level of detail necessary for
replication) of the robot or robots
used in the experiment.
It is important that the code and
HW are well identified to be able to
replicate experiments. This might be

1) Is it an experimental paper?
2) Are the system assumptions/hypotheses clear?
3) Are the evaluation criteria spelled out explicitly?
4) What is being measured and how?
5) Do the methods and measurements match the criteria?
6) Is there enough information to reproduce the work?
7) Do the results obtained give a fair and realistic picture of the system
being studied?
8) Are the drawn conclusions precise and valid?
Figure 3. The Euron GEM guidelines in a nutshell [1]. The guidelines were meant as an
adaptation of the basics of the scientific method to robotics and AI.

180

IEEE ROBOTICS & AUTOMATION MAGAZINE

SEPTEMBER 2017

The R-Article Life Cycle
As we aim to foster the spread of a reproducibility culture in our community, the
introduction of the R-article is not
enough. We will provide a venue for publishing reproduced experiments (this
could be a nice opportunity for Ph.D.
students, but not only for them). It will be
possible to publish a short article about
the results replication of an R-article-
what we might call an r-article. Such articles will be peer reviewed like any other
RAM article and will undergo a quick
data and code consistency check. Similarly, the authors of the original R-article
will be able to submit, again, in the form
of a short peer-reviewed article, a reply
to the authors of the r-article, again, with
a data and code consistency check. We
will call this a Reply article.
The outcome will be a two-stage,
high-quality review process. The first
stage will be little more than the already
very rigorous RAM review process. The
second, more accurate stage will be the
reproduction of the results by the community. We think the process will
improve over time as we adapt and
expand the data and code-sharing platform, integrate other tools, and refine
the procedure. As the process is implemented, we will learn from experience-and, of course, comments and
suggestions are welcome. Feel free to
contact me at fabio.bonsignorio@
santannapisa.it or the editor-in-chief
at ieeerameic@gmail.com if you need
any further clarification.
Call for R-Articles
From this issue's publication date, we
will begin to accept R-articles for RAM.
We will start to publish them as soon as
the first accepted R-article comes out of
the review pipeline. We are proud to say
that we are the first top-tier robotics
publishing venue to do this, and we sincerely hope that many others will follow. It will be a learning process for
sure, but we begin on a solid foundation
of wide discussion and strong efforts in
the community in the past years. Please
help us to make history! Let's start!

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2017