IEEE Robotics & Automation Magazine - December 2015 - 126

This article presents the beginnings of a framework for
robotic hand performance benchmarking. Many of the concepts presented are the results of an informal working group
organized by the National Institute of Standards and Technology (NIST) that continues as part of the recently formed IEEE
Robotics and Automation Society (RAS) Robotic Hand Grasping and Manipulation (RHGM) Technical Committee (IEEE
RAS Technical Committee on Robotic Hand Grasping and
Manipulation; http://www. rhgm.org). A successful grasp is the
combination of an appropriate coupling of hardware (i.e., robot
hands) and algorithmic components (e.g., grasp planners and
grasp control). Both sides have to be benchmarked and the
assessment process needs to be repeatable. This article introduces a subset of hardware and control benchmarks that are
demonstrated using a set of robotic hand platforms. This particular set of hands was chosen based on their availability, and
more hands will be tested by NIST as they become available. A
parallel effort aimed at defining best practices for benchmarking
and comparing grasp planning algorithms is also being developed but will not be discussed here due to space limitations.
The material presented in this article is meant to propose a
path to develop replicable performance measures for robotic
grasping but is not intended as the definitive methodology.
More details regarding the actual tests and data analysis are
available at http://www.nist.gov/el/isd/grasp.cfm. It is foreseeable and desirable that the core ideas presented in this article
will be extended through a community driven approach. To
the best of our knowledge, no comparable effort has been formulated in the past, and this is the first article detailing a com-

prehensive methodology for repeatable research in the robotic
hand technology that spans both hardware and software components. Repeatability in grasping or any other robotic subdomain requires conscious commitment by the researchers to
share information (designs, data, models, code, and so on) in
an open and understandable format-a mentality we hope to
inspire with the ideas presented here.

Grasp Performance Tests
Here we present the physical measurements for assessing performance of robotic hands using measurement techniques
external to the system under test. Physical results of grasping
are reported using both qualitative and quantitative data.
Qualitative measures are easily found in the robotic grasping
research literature; however, examples of applying quantitative
measures to evaluate grasp performance are sparse and have
only been developed formally for prosthetics [7]-[11].
When evaluating the capabilities of a robotic hand, performance tests should be agnostic to the other system components,
such as the robot arm and the perception system. While it is
possible to access data directly from a robotic hand and to derive
the defined metrics, these measurements would be based on the
inherent properties of the system under test. Therefore, independent measurement systems must be developed to support
testing to allow for comparative metrics between systems to
establish extrinsic ground truths.
Breaking down a problem into its parts can provide novel
insights toward its solution. In particular, consider the underlying tasks associated with a robotic pick-and-place operation
for a fully integrated multifingered robotic
hand, as shown in Figure 1. This concept
was introduced within the NIST-organized
grasp metrics informal working group by
Grasp Planning
(No Object)
SynTouch LLC. The terminology chosen
here was based on that used throughout the
Problem
Problem
Noncontact
community and is not the result of any sinObstruction
Phase
Obstruction
gle author. Each task in this particular operClear
Cage
Clearance
Clearance
ation possesses a number of associated
Size
problems that can serve as a basis for
extracting performance measures. FurtherContact Point
Movement
Release
Constrain
Clearance
more, identifying the significance of particStability
Synchronization
ular performance measures for different
Grasp Phase
grasping tasks would provide valuable
Slippage
knowledge on necessary functionalities
Slippage
Crusing
Unload
Load
Ejection
toward task completion.
Ejection
Robotic hands are an integrated system
of mechatronics, sensors, and control
Slippage
Slippage
Place
Pick
Compliance
Crushing
algorithms with considerable variability in
Ejection
Ejection
Manipulation
their number of degrees of actuation,
Phase
degrees of freedom (DOF), and joint
types. Furthermore, a variety of touchSlippage
In-Hand and Arm Manipulation
Compliance
sensing strategies across different plat(Hold Object)
Ejection
forms exists. Advanced sensing capabilities include fingertip embedded six-axis
load cells [12], pressure-sensitive tactile
Figure 1. The pick-and-place task segmentation and transitions between grasped
sensors, vision-based contact sensing
and ungrasped states, and examples of potential problems.
126

IEEE ROBOTICS & AUTOMATION MAGAZINE

DECEMBER 2015

http://www http://www.rhgm.org http://www.nist.gov/el/isd/grasp.cfm

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2015