IEEE Robotics & Automation Magazine - September 2015 - 4

from the editor's Desk

Research Reproducibility and Performance
Evaluation for Dependable Robots
By Eugenio Guglielmelli

his issue of IEEE Robotics &
Automation Magazine (RAM)
focuses on reproducibility and
measurability of robotics research. When announcing this special
issue, I already outlined the relevance
of the challenge for our community, as
also emphasized by our cover page.
Please do not expect to find in the features of this issue news on recent discoveries about basic physics principles
enabled by (humanoid) robots; what
the issue is actually about is the development of methods, tools, benchmarks,
data sets, standard setups for performance evaluation via experimental
tests-or even competitions-devised
for a variety of robotics research areas. I
am very happy that we had a high
number of submissions with many
high-quality contributions to such an
extent that a focused section of the next
issue will be devoted to these same topics. It is also remarkable that the IEEE is
currently considering research reproducibility as becoming the key quality
requisite for articles appearing in IEEE
publications. This idea was recently debated at the IEEE Panel of Editors
meeting which was held in Washington
D.C. last April. Gianluca Setti, the IEEE
past vice-president for publications,
who promoted such debate within the
IEEE, kindly agreed to be interviewed
to give his perspective on the relevance
of this topic, as reported in the Turning
Point column of this issue (see page
192). Thanks to this successful special
Digital Object Identifier 10.1109/MRA.2015.2470815
Date of publication: 11 September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

issue, the IEEE Robotics and
Automation Society demonstrates that we are well aware
of and perfectly in line with
this important trend. This is
clearly a key aspect for a mature
research field like robotics and automation, which is expected to deliver more
disruptive innovation in the short-medium term. But it is definitely not
enough. Early July 2015 an accident
was reported in an automotive factory
in Germany: A worker died while setting up a robotic welding unit. Immediately after that this sad event was
reported in the news, there was some
speculation on media channels, amplified by social networks, about the possibility that this was the first accident
related to the use of a novel generation
of collaborative robotic systems designed to allow the active presence of
human workers within the robot workspace during operations. Fortunately, it
was soon clarified that this was not the
case, but I believe that this reaction is
clear evidence of the level of attention
paid by the general audience to the
risks related to the introduction of robots in our daily living scenarios. That's
why what I really consider as the most
important payoff of having measurable
and reproducible results while evaluating the performance of robotic systems
is the possibility of demonstrating not
only their safety but also other crucial
properties, such as dependability and
resilience, which robotics technology
shall feature to be eventually accepted
by nonexpert users into novel application scenarios. The concept of depend-

September 2015

ability was introduced for
computer systems in 1992
by the late Dr. Jean Claude
Laprie, a senior researcher at
the Laboratory for Analysis
and Architecture of Systems
(LAAS), Toulouse, France. The idea
that modern robots also should be designed to be dependable was first proposed by the late Dr. Georges Giralt, an
IEEE Fellow and a pioneer of robotics
research also based at LAAS, who initiated a successful series of workshops in
2001 and edited an IEEE RAM special
issue (vol. 11, no. 2, 2004) on this topic.
Dependability means not only safety,
reliability, service availability, integrity,
confidentiality, and maintainability-
all properties that can be objectively
measured-but also the level of trust
end-users have in the system performance: How far would you go in depending on a service directly provided
to you by a robot system, such as an
autonomous car, a drone, a coworker
or a personal assistant for independent
living? Tackling such subjective perspective for evaluating the system performance still requires a lot of research
effort, especially when you consider
that robots should be capable not only
of being dependable in routine operational conditions, but also resilient to
external malicious attacks, which purposely aim at generating failures and
accidents. No doubt, it is an intriguing
perspective that will engage generations of researchers. Enjoy the issue!

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015