IEEE Robotics & Automation Magazine - June 2011 - 50

beyond that offered by other data sets that could be
extracted and used for algorithm training include the
appearances of people in multiple robot sensors, typical
human poses in office environments (e.g., sitting and
standing), illumination conditions (e.g., heavily back-lit
offices with windows), scene features (e.g., ceilings, desks,
and walls), and how people move around the robot. This is
just a small sample of the applications for this data set.

(a)

(b)

Figure 5. The Mechanical Turk interface for annotating outlines
of people for the Moving People, Moving Platform data set. (a)
Workers were presented with the original image with a
bounding box annotation of one person (by another worker),
and (b) an enlarged view of the bounding box on the right. The
worker drew a polygonal outline of the person in (b). (Photo
courtesy of Willow Garage, Inc.)

failing to notice a malicious grader leads to numerous
misgraded annotations. These facts encouraged us to
grade the annotations personally and write lengthy comments to workers making consistent mistakes. The workers were extremely receptive to this approach, quickly
correcting their mistakes, thus significantly reducing
duplication of work. Overall, personalized feedback for
the small number of workers reduced our own workload.
There are other ways to identify incorrect annotations;
however, they were not applicable in this situation. For
example, the completely automated public Turing test to tell
computers and humans
* apart (reCAPTCHA) style
[16] of presenting two anAnnotation of the data set
notations and grading
the second based on the
was crowd-sourced using
first assumes that the
errors are consistent. For
Amazon's Mechanical Turk the annotation task in the
Moving People, Moving
marketplace.
Platform data set, how* ever, errors resulted
from misunderstanding
the instructions for a particular image scenario (e.g., a person truncated by the image border). Unless both of the
images presented contain the same scenario(s), the redundancy of having two images cannot be exploited.
Applications
This data set is exclusively intended for offline training and
testing of person detection and tracking algorithms from a
robot perspective. The use of multiple sensor modalities,
odometry, and situational information is encouraged.
Some possible components that could be tested using this
data set are face detection, person detection, human pose
fitting, and human tracking. Examples of information
50

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2011

Future Work
It is important to take a moment to discuss the possible constraints on algorithm design imposed by the annotation format and methodology. Two-dimensional (2-D) outlines can
only be accurate in the image orientation and resolution.
Robots, however, operate in three dimensions. Given that
stereo camera information is noisy, it is unclear how to
effectively project information from a 2-D image into the 3D world. The introduction of more reliable instantaneousdepth sensors may ameliorate this problem. However, even
a device such as the Microsoft Kinect sensor [17] is
restricted to one viewpoint. Algorithms developed on such a
data set can only provide incomplete information. A format
for 3-D annotations that can be obtained from an untrained
workforce is an open area of research.
Short-term work for this data set will be focused on
obtaining additional types of annotations. It would be
informative to have semantic labels for the data set such as
whether the person is truncated, occluded, etc., and pose
information such as whether the person is standing, sitting,
etc. Future data sets may focus on perceiving people during
interaction scenarios such as object handoff. Additional
data from new sensors, such as the Microsoft Kinect,
would also enhance the data set.
Finally, an additional interesting data set could be constructed containing relationships between people and
objects, including spatial relationships and human grasps
and manipulations of different objects. Object affordances
could enhance the other data sets described in this article.
The Household Objects and Grasps Data Set
A PR's ability to navigate around and interact with people
can be complemented by its ability to grasp and manipulate objects from the environment, aiming to enable complete applications in domestic settings. In this section, we
describe a data set that is part of a complete architecture
for performing pick-and-place tasks in unstructured (or
semistructured) human environments. The algorithmic
components of this architecture, developed using the ROS
framework, provide abilities such as object segmentation
and recognition, motion planning with collision avoidance, and grasp execution using tactile feedback. For more
details, we refer the reader to our article describing the
individual code modules as well as their integration [18].
The knowledge base, which is the main focus of this article,
contains relevant information for object recognition and
grasping for a large set of common household objects.

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2011