IEEE Robotics & Automation Magazine - June 2011 - 61

object, a bottle of instant pancake mix, was found on germandeli.com, a shopping Web site, and a recognition
model was created from the product picture (described in
detail in the "Object-Recognition Models" section and
shown in Figure 4).
Once the robot knows which objects it needs and how
they look like, it has to find them in the kitchen environment. Since it has not only downloaded the picture of the
pancake mix but also the information about its properties,
it can infer that it needs to be stored in the refrigerator.
Figure 5 illustrates the inference steps that are performed.
The reasoning process combines encyclopedic knowledge
about the fridge [Figure 5(a)], commonsense knowledge
that a refrigerator is the storage place for perishable goods,
spatial knowledge describing an instance of a refrigerator
at a certain location in the environment, and knowledge
about the pancake mix that was automatically generated
from an online shop's Web site (see the "Object Properties" section). To combine these different kinds of knowledge from different sources, a common representation is
crucial. In our system, all knowledge is formally represented in the KnowRob knowledge base [35].
The methods presented in this article helped the robot
to acquire important information. The methods for acquiring the knowledge worked well; however, the major problem is the lack of knowledge sources that provide the
required information. We pushed the automatic import
from existing sources as far as we could but still had to
manually add some scenario-specific routines or knowledge. For example, the robot's knowledge about actions,

PhysicalIDevice
Is a
HouseholdAppliance

is-a
Is a

Is a

ElectricalHouseholdAppliance RefrigeratedStorageDevice
Is a

Is a

GermanDeliobject

StorageConstruct

Is a

objects, and processes is still not deep enough to competently understand the instructions written for humans.
The instructions in the experiment refer to the pancake
mix, a liquid stufflike substance (i.e., something that can be
divided into smaller pieces without changing its type).
Although the robot can map this description to the correct
concept in its knowledge base, it would also have to infer
that this stuff is usually in a container and it thus has to
search for this container, open it, pour the mix onto the
pancake maker, and further estimate how much of the mix
is required. These relations are still manually specified
since there is no source of information providing such
knowledge. Equipping robots with sufficiently deep

ContainerArtifact

Is a

CoolingDevice

Figure 4. Picture of a bottle of pancake mix obtained from an
online shop. (Photo reprinted with permission from
www.germandeli.com.)

Is a
FoodIngredientOnly

Perishable

Box-Container
Is a

Is a

Storage Place for

Is a Groceries

Is a

Is a
Groceries_Aisle_1

MixForBakedGoods

Refrigerator

Is a
PancakeMix

Instance of

Is a

Baking_Mixes
Is a

Is a

Pancake_Waffle_Muffin_Mixes
Is a
Mondamin_P fannkuchen_Teig_Mix_200g_Pancake_Mix

Instance of

Figure 5. Reasoning steps to infer a probable storage location for a bottle of pancake mix.

JUNE 2011

IEEE ROBOTICS & AUTOMATION MAGAZINE

http://www.mandeli.com http://www.germandeli.com

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2011