IEEE Robotics & Automation Magazine - September 2015 - 40

includes a wide range of objects to span many different manipulation applications.
We provide a detailed overview of prior related benchmarking efforts, discussing their scope and limitations. For organization purposes, we first discuss work primarily related to robotic
manipulation (including vision and learning applications), then
efforts in rehabilitation, including prosthetics.
Robotic Manipulation
The necessity of manipulation benchmarks is highly recognized in the robotics community [12]-[14] and continues to
be an active topic of discussion at workshops on robotic manipulation (see [15]). As mentioned earlier, the majority of
prior work related to object sets has involved just object images and models (with varying degrees of information, from
purely shape information to textural plus shape). Such work
has often been created for research in computer vision (see
[2], [16], and [17]). There have also been a number of shape/
texture sets designed for/by the robotics community, particularly for applications such
as planning and learning.
The object set is specifically The Columbia grasp database [3] rearranges the
designed to allow for
object models of the
Princeton shape benchwidespread dissemination
mark [18] for robotic manipulation and provides
of the physical objects and mesh models of 8,000 objects together with a
manipulation scenarios.
number of successful
grasps per model. Such a
database is especially useful for implementing machine-learning-based grasp synthesis algorithms in which large amounts of labeled data are
required for training the system. A multipurpose object set,
which also targets manipulation, is the Karlsruhe Institute of
Technology (KIT) object models database [19] which provides stereo images and textured mesh models of 100 objects.
While there are a large number of objects in this database,
the shape variety is limited, and like the previously mentioned data sets, the exact objects are typically not easily accessible to other researchers due to regional product
differences or variation over time, and they generally cannot
be purchased in one place as a set.
There have only been two robotics-related efforts in
which the objects are made relatively available. The household objects list [6] provides good shape variety that is appropriate for manipulation benchmarking as well as a
shopping list for making the objects more easily accessible to
researchers. Unfortunately, the list is outdated, and most objects are no longer available. The three-dimensional (3-D)
models of objects in [6] are not supplied, which prevents the
use of the object set in simulations. Very recently, the Amazon Picking Challenge [7] also provides a shopping list for
items, but those were chosen to be specific to the bin-picking
application and do not have models associated with them.
40

IEEE ROBOTICS & AUTOMATION MAGAZINE

September 2015

In terms of other robotic manipulation benchmarking efforts, a number of simulation tools have been presented in the
literature. The OpenGRASP benchmarking suite [20] presents a simulation framework for robotic manipulation. The
benchmarking suite provides test cases and setups and a standard evaluation scheme for the simulation results. So far, a
model-based grasp synthesis benchmark has been presented
using this suite. VisGraB [22] provides a benchmark framework for grasping unknown objects. The unique feature of
this software is its utilization of real stereo images of the target
objects for grasp synthesis as well as execution and evaluation
of the result in a simulation environment. For gripper and
hand design, benchmark tests [39], [40] are proposed for
evaluating the ability of the grippers to hold an object, but
only cylindrical objects are used.
Prosthetics and Rehabilitation
In the general field of rehabilitation and upper-limb prosthetics, there are a number of evaluation tools used by therapists to attempt to quantify upper-limb function in humans.
Some of these are commercially available, clinically verified,
and have been substantially covered in the literature, including normative data to compare a patient's performance to
baselines. While some tools are commonly used, other tests
have only been proposed in the literature and not (yet, at
least) been widely utilized. Many of these tests aim to evaluate the ability of patients to perform tasks that contribute to
activities of daily living.
The tests that are commercially available are the box-andblocks test [41]; the nine-hole peg test [42]; the Jebsen-Taylor
hand-function test [11]; the action research arm test (ARAT)
[10]; the graded redefined assessment of strength, sensibility,
and prehension (GRASSP) test [9]; and the Southampton
hand-assessment procedure (SHAP) [8]. The setups for the
box-and-blocks and nine-hole peg tests are very specific, with
evaluation based on timed movements of simple objects. The
setup for the Jebsen-Taylor hand-function test includes objects
for manipulation actions, such as card turning, and moving
small (paper clips, bottle caps), light (empty cans), and heavy
objects (1-lb weighted cans), but it utilizes a small number of
objects of limited shape and size variety. The ARAT assesses
upper-limb function, and its commercial set [43] contains objects such as wooden blocks of various sizes, glasses, a stone, a
marble, washers, and bolts. The test proposes actions like placing a washer over a bolt and pouring water from one glass into
another. The GRASSP measure has also been proposed for the
assessment of upper-limb impairment. It is based on a commercial kit available in [44]. Apart from a specialized manipulation setup, the kit also includes the nine-hole peg test, jars,
and a bottle. The SHAP setup includes some objects of daily
living, such as a bowl, a drink carton, and a jar, together with
some geometrical shapes. Patients are requested to perform a
variety of manipulation tasks, mostly involving transporting
objects but also including pouring a drink, opening the jar,
and so on. Considering manipulation benchmarking in robotics, the box-and-blocks, nine-hole peg, and Jebsen-Taylor

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015