IEEE Robotics & Automation Magazine - December 2015 - 125

GALILEO - IMAGE LICENSED BY INGRAM PUBLISHING,
LAMP-©ISTOCKPHOTO.COM/JONNY JIM

FOCUSED SECTION ON
REPLICABLE AND MEASURABLE
ROBOTICS RESEARCH

Grasping the
Performance
Facilitating Replicable Performance Measures
via Benchmarking and Standardized Methodologies

ALL-FREE-DOWNLOAD.COM

By Joe Falco, Karl Van Wyk, Shuo Liu,
and Stefano Carpin

" t is comparatively easy to make computers exhibit adult-level
performance on intelligence tests or playing checkers, and
difficult to give them the skills of a one-year-old when it
comes to perception and dexterity" [1]. More than 15 years
after it was first stated, Moravec's paradox still holds true today.
Fueled by vigorous research in machine learning, the gap has
consistently narrowed on the perception side. However, most of
the fine manual motor skills displayed by a toddler are, to date,
far beyond what robots can do. It is true that many valuable
tasks involving physical interaction with objects can be solved by
contemporary robots as indicated by a thriving industrial
robotics sector. However, in the future, robots are expected to
work side by side with humans in unstructured environments,
and the ability to reliably grasp and manipulate objects used in
everyday activities will be an unavoidable requirement. Today's
robots are far from being ready for this challenge.
Multiple reasons account for this persistent gap-some are
technological, whereas others are methodological. We postulate
that one of the major obstacles inhibiting progress in this area
Digital Object Identifier 10.1109/MRA.2015.2460891
Date of publication: 10 December 2015

surrounds the continued inability to replicate and compare
results generated by the grasping community. As in other robotics subdomains, too many researchers lack the ability to contrast
their findings with those of their peers using a principled,
accepted methodology. With a surge of robotic hand designs
[2]-[5] and algorithmic developments, it is difficult to make
informed decisions or draw quantitative conclusions regarding
commercial products or lab prototype performances in various
task-related settings.
Standardized performance testing is an emerging tool within
the robotics community that is proving itself worthy and offers
unbiased evaluation methods to assess how well a system performs a particular ability [6]. The results of such evaluations and
benchmarks help match capabilities to end-user needs and provide developers insight for improving their product designs.
Accordingly, we created a subset of grasp performance metrics,
test methods, and example experiments to be vetted by the
robotic hand research community. Together, accelerated progress can be made in this critical area that will lead to a set of standard measures and performance test methods. Such standards
will guide the development of future grasping technologies and
allow for seamless system benchmarking.

1070-9932/©2015 Translations and content mining are permitted for academic research only. Personal use is also permitted, but replication/
redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

DECEMBER 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

125

http://www.ALL-FREE-DOWNLOAD.COM http://www.ieee.org/publications_standards/publications/rights/index.html

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2015