IEEE Robotics & Automation Magazine - September 2017 - 110

making plans to fulfill them. Furthermore, human-scale
everyday manipulation tasks require a reasoning component
for making inferences based on available knowledge. Note
that the architecture in Figure 2 is presented to visualize the
mentioned components for clarity. As indicated in the section "System Integration," such a strong modularization may
be questioned based on the biological basis, and thus is not
the only possible organization.
This article reviews the recent work and points out the
open research directions related to knowledge-enabled reasoning, perception for world modeling, planning and acting,
learning, and human-robot collaboration in the context of
daily manipulation. Due to the lack of a unified benchmark,
it is not possible to provide a comparative analysis of the
reviewed approaches presented under different settings.
Nonetheless, we believe that this survey will provide a useful
reference for researchers working in related research areas
by giving an overall picture of the current progress toward
building cognitive robots handling manipulation tasks in
human environments.
Knowledge-Enabled Reasoning
Daily manipulation tasks require robots to possess a considerable amount of knowledge about a large variety of objects and
their affordances (i.e., possible actions on these objects) in different human settings (e.g., cabinets, tables, shelves, etc.). Efficient knowledge-processing skills are needed to make
decisions based on this knowledge in real time. This knowledge should be kept in a
knowledge base for formalizing self-capabilities,
Despite important
goals, task-relevant and
contextual information,
achievements in
and the state of the environment by keeping a
semicontrolled settings,
useful balance of both expressiveness and tractamobile manipulation in
bility. Different logical and
probabilistic formalisms
unstructured and open
can be considered for
knowledge representaenvironments is still an
tion and reasoning de pending on the nature of
unsolved problem.
the knowledge to be represented [9]. Moreover, the
robot's knowledge needs to be associated with real-world data
acquired through sensing and maintained continually as the
robot operates in its environment. This is known as the symbol grounding problem, for which an overview of recent studies is given in [10]. The ability to use common sense for
guiding decisions and figuring out missing information
required for performing given tasks is also an important key
to success [11].
The aforementioned issues are addressed in some recent
works [12]-[16]. For example, KnowRob [12] is a knowledge-processing infrastructure that provides service robots
110

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

SEPTEMBER 2017

with the knowledge needed for performing everyday manipulation tasks by complementing ontological knowledge with
common sense. It is based on a virtual knowledge-base paradigm where data abstraction is performed on demand based
on the most up-to-date information, and it provides a hybrid
inference system combining logical and probabilistic inference with an interface for external reasoning methods.
OpenRobots Ontology (ORO) [13] is another knowledge
management platform designed for robots working in conjunction with humans and defines concepts useful for
tasks involving human-robot interaction, and it maintains
independent cognitive models for each agent in interaction,
enabling reasoning based on beliefs from each agent's perspective. Some studies demonstrate the use of knowledgebased approaches for different related aspects with service
robots, such as maintaining perceptual-symbolic correspondences about physical objects in an intelligent home environment [14]; providing a unified interface for performing
visual inference about objects, actions, and affordances [15];
and facilitating learning from experience on object manipulation episodes [16].
The knowledge base of a service robot can be populated
by a wide range of information sources. For example, Ji and
Chen [17] propose an approach based on causal theories for
service robots to determine missing knowledge required for
accomplishing given tasks and then to acquire this knowledge from the Open Mind Indoor Common Sense (OMICS)
database [18]. Some other recent studies investigate how
robots can utilize information from Internet resources to
acquire step-by-step instructions for routine tasks [19], general common sense about objects and actions for automated
planning [20], object models for recognition and grasping
[19], [21], and background knowledge about human environments relating objects to their possible locations [22].
Envisioning an exclusive worldwide network for robots,
RoboEarth [23] is proposed as a web community for robots
to autonomously share task-related information such as task
descriptions, models of robot components, object models for
recognition, and environment maps. RoboBrain [24] pushes
these studies forward by introducing a large-scale knowledge
engine that continuously learns multimodal knowledge from
different sources like natural language texts, images, and videos to provide robots with the knowledge required for performing their tasks.
Open Problems and Future Directions
Despite the notable progress in recent years, research in this
area is still needed to realize robots sufficiently knowledgeable in routine manipulation tasks. First of all, it is necessary
to decide on the contents of the knowledge to be encoded
beforehand and the knowledge that is expected to be
acquired by the robots during their operation. Because it is
not feasible to preprogram robots for all possible contingencies they may face in human environments, a considerable
amount of knowledge needs to be figured out by the robots
themselves. Moreover, interacting with humans requires
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2017
