IEEE Robotics & Automation Magazine - June 2011 - 65

(a)

(b)

Figure 8. (a) Samples from test and (b) training data for the classes Volvic and Coca Cola. (Pictures courtesy of The Coca Cola
Company, Kraft Food Inc, Danone Waters.)

CAD Model Databases
To manipulate objects, the robot does not only need to
know their types, but it also needs spatial information. The
CAD models are one of the most accurate descriptions for
rigid objects. Once a CAD model is available, a robot can
recognize and localize the corresponding object in the
environment and plan its manipulation actions.
The problem is that a robot usually does not have models for all objects it needs to manipulate. Assuming that the
class of the object is known, like pan or cup, we propose to
download a set of CAD models from the Internet and to
find the model that best fits the current sensor data (see [8]
for details). The method requires that the system has a
rough estimate of the location of the object (e.g., on top of
the stove or inside a specific cupboard) and a good approximation of its size, from a segmentation of 3-D sensor data
or from commonsense knowledge.
To make use of CAD models from the Internet for recognizing objects, a robot first has to select the most relevant
objects to be downloaded and then needs to verify and compare the appearance of the actual object with the CAD model.
The problem of finding relevant models is tackled using tags,
which are assigned to the models by search engines such as
the Google 3-D Warehouse. As part of the search results for
words such as pan or cup, we get a list of tags that allow to
put the results into the right context. Usually, those tags
describe the content of the CAD model better than its title, or
the fact that it was returned as an answer to a query. Models
are selected from the result set if their tags have a low semantic distance to the search term [25]. This semantic distance is
computed using the WordNet lexical database and describes
how close the words are in the WordNet taxonomy. All
selected models can be matched to the current scene to get a
certain score based on the current sensor data and the respective method. If there is more than one good match, the system can interpolate between two 3-D models to get an
intermediate model that may fit the data even better than the

original ones. This morphing procedure is explained in detail
in [39] and an example is shown in Figure 9.
Once a model is selected, the system has to decide
whether this CAD model fits the sensor data from the initial
position estimate. We integrated two methods into our system that are capable of taking an arbitrary CAD model and
comparing it with the sensor data of the expected position
of the real object. The first method is based on geometric
edges of the CAD model that are projected to a two-dimensional (2-D) image. Possible object poses are precalculated,
and the model is projected into a virtual edge image and
compared in an efficient way to the image of the scene. This
search is called 3-D shape-based matching and uses as core
the method proposed by Ulrich et al. [37]. This method is
evaluated in our setup with some improvement in [7]. The
second method relies on 3-D sensors. Instead of precalculating edges in the image, it computes the normals and the

(a)

(b)
Figure 9. (a) The original model and an interpolated model are
applied to the same scene to show that the interpolated model
fits better in this case. (b) Two models from the search for cup
and a morphing between the two cups are displayed.

JUNE 2011

IEEE ROBOTICS & AUTOMATION MAGAZINE

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2011