IEEE Robotics & Automation Magazine - September 2015 - 50

Box-and-Blocks Test
As mentioned in the "Related Work" section, the box-andblocks test [41] is a widely used assessment technique that is
utilized in prosthetics and rehabilitation fields. The test evaluates how many blocks can be grasped and moved from one
side of the box (Figure 18) to the other in a fixed amount of
time. We believe that the application of this test can also be
quite useful for assessing the manipulation capabilities of robots. To establish a baseline performance for this test for robotic manipulators, we applied the box-and-blocks test with
a PR2 robot (Figure 18) by implementing a very simple heuristic rules. The robot picks a location from a uniform distribution over the box and
attempts to pick up a
block. The gripper's pose
The OpenGRASP
aligns with the length of
the box. The gripper is
benchmarking suite [20]
then closed and checked
if it is fully closed. If the
presents a simulation
gripper closes fully, this
means no blocks have
framework for robotic
been grasped and, therefore, the robot chooses a
manipulation.
new location to attempt
another pick. The robot
repeats this heuristic until the gripper is not fully closed.
When a grasp is detected, the robot moves to the destination
box and releases the block. By using this heuristic, we run ten
experiments of 2 min each and report the results at [64].
Conclusions and Future Work
This article proposes a set of objects and related tasks as
well as high-resolution scans and models of those objects,
intended to serve as a widely distributed and widely utilized
set of standard objects to facilitate the implementation of
standard performance benchmarks for robotic grasping
and manipulation research. The objects were chosen based

Figure 18. The PR2 executing the box-and-blocks test.

IEEE ROBOTICS & AUTOMATION MAGAZINE

September 2015

on an in-depth literature review of other object sets and
tasks previously proposed and utilized in robotics research,
with additional consideration to efforts in prosthetics and
rehabilitation. Furthermore, a number of practical constraints were considered, including a reasonable total size
and mass of the set for portability, low cost, durability, and
the likelihood that the objects would remain mostly unchanged in years to come. High-resolution RGB-D scans of
the object in the set were completed, and 3-D models have
been constructed to allow easy portability into simulation
and planning environments. All of these data are freely
available in the associated repository [55]. Over the course
of 2015, 50 objects sets will be freely distributed to a large
number of research groups through workshops/tutorials associated with this effort. Additional object sets will be made
available to purchase otherwise.
While a common set of widely available objects is a
much-needed contribution to the manipulation research
community, the objects themselves form only part of the
contribution of the YCB set. The generation of appropriately detailed tasks and protocols involving the objects is ultimately what will allow for replicable research and
performance comparison. We make inroads into that problem in this article by proposing a structure for protocols
and benchmarks, implemented in a template as well as six
example protocols. We hope that specification of protocols
and benchmarks will become subcommunity driven and
continually evolving. Specific aspects of manipulation and
other specific research interests will naturally require different task particulars (i.e., specified and free parameters). We,
therefore, plan to involve the research community in this
effort via our web portal [63]. We will work toward having
the majority of such protocols come from the user community rather than the authors of this article. In addition, we
plan to have on this portal a records-keeping functionality
to keep track of the current world records for the different
tasks and protocols, along with video and detailed descriptions of the approaches utilized, generating excitement,
buzz, motivation, and inspiration for the manipulation
community to compare approaches and push forward the
state of the art.
Other efforts that we plan to undertake include more detail about the objects proposed, including information about
the inertia of the objects, as well as frictional properties between the objects and common surfaces. Additionally, we
will expand our treatment of the modeling of the objects, including addressing the tradeoffs between number of triangles
in a mesh and the reliable representation of the object geometry. Furthermore, before final publication and distribution
of the object set, we will seek additional input from the research community on the specific objects in the set.
It is our hope that this article will help to address the longstanding need for common performance comparisons and
benchmarks in the research community and will provide a
starting point for further focused discussion and iterations on
the topic.

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015