IEEE Robotics & Automation Magazine - September 2017 - 113

different perception algorithms with complementary
strengths in a knowledge-enabled approach [28], maintaining a consistent world model from noisy observations
[44]-[46], utilizing domain knowledge for lifelong object
discovery [42], and using contextual information for guidance [39]-[41]. These are promising avenues for future
research, but they should be investigated further in terms
of scalability for a variety of routine manipulation tasks.
Recent successful applications of deep learning algorithms
in computer vision [49] have opened another direction of
research toward increasing perceptual capabilities of
robots working in unstructured environments. Everyday
manipulation tasks also necessitate interactions with novel
objects without models for recognition [29], as it is not
feasible to manually provide models for all objects that
may be encountered in human environments. Figuring out
kinematic properties of these objects [30] and detecting
their functional affordances [32]-[34] are important prerequisites for enabling their usage in object manipulation
tasks. More research efforts will go to enable service robots
to continuously monitor their environments in real time
while effectively handling daily manipulation tasks.
Planning, Acting, and Adaptation
A useful service robot must be able to make plans and adapt
them for completing manipulation tasks with dexterity in
human environments. These plans need to be executed by
taking into account the geometric constraints imposed by the
environment (e.g., obstacles) and the actuation capabilities of
the robot. Therefore, integration of high-level symbolic planning and reasoning with low-level perception and control is
crucial for performing object manipulation tasks effectively in
unstructured environments.
Many studies concentrate on finding a proper integration
method for task and motion planning [50]-[56]. Hierarchical Planning in the Now (HPN) [50] deals with this problem by continuously interleaving task planning on the basis
of hierarchical decomposition and motion planning that
relies on fast approximate geometric computations. HPN
has been extended by introducing monitoring and replanning to ensure correct action execution and by performing
planning in the belief space to account for state estimation
uncertainties [51]. A planner-independent interface layer for
combining task and motion planners has been proposed
[52], which relies on logical predicates to communicate reasons for failures during geometric search to the task planner
to update the symbolic state for replanning. Another
approach is based on automatically generated constraints
from a given symbolic action sequence and the kinematic
model of the robot to reduce the state space during geometric
evaluation in manipulation planning [53]. Another study
based on interleaved backtracking relies on a geometric task
planner [57] rather than a motion planner to make decisions
(e.g., on what grasp to use) in the geometric level [54]. The
Knowledge of Volumes framework [55] integrates the Planning with Knowledge and Sensing planner [58], which is able

to deal with incomplete information by using sensing actions
to gather information when necessary, with motion planning
based on detecting collisions of volumes representing objects
and the robot's parts. The feasibility of learning a proper mapping between symbolic and geometric levels by simulating
executions of example plans and using the learned mapping
for manipulation planning is shown in [56].
Some studies concentrate more on reasoning required for
planning and acting [59]-[61]. For example, the Cognitive
Robot Abstract Machine [59] is a system combining an
expressive planning language with reasoning mechanisms
enabling robots to reason and modify their control programs.
In this framework, control decisions are formulated as runtime inference tasks using the robot's knowledge base to create general, flexible, and reusable plans. Common sense
reasoning and probabilistic planning [60] integrates partially
observable Markov decision process-based planning with
common sense reasoning using P-log [62], which is a
probabilistic extension of Answer Set Programming [63] for
guiding planning in partially observable environments with
common sense. Following the argument [64] that reasoning
and deliberation required for acting are underestimated in the
planning research literature, [61] presents a preliminary
model toward developing a formalization suitable for
real-world acting by combining continual online planning
with deliberative acting in a unified hierarchical refinement
method representation.
Monitoring the execution process is also very important
for a service robot to detect anomalies and to adapt actions
accordingly for successful task completion. Some recent
studies address this issue
[65]-[67]. The integrated mobile manipulation
Interpreting the world
and grasping approach
presented in [65] combines
from multimodal sensory
real-time environment
modeling, navigation,
data acquired from the
grasp planning, and reactive control (Figure 5). This
environment is essential
approach relies on reactive
grasp behaviors based on
for service robots working
tactile sensory data to deal
with problems caused by
in unstructured and
inaccurate perception and
on active monitoring duropen environments.
ing motion execution to
deal with unforeseen obstacles. In another work, a reactive reaching and grasping system
is proposed in an effort to realize a vision-guided system for
object manipulation in unstructured environments [66]. The
proposed system tightly integrates perception and control to
realize basic hand-eye coordination on an iCub humanoid
robot [68], enabling it to adapt its behaviors to changes in its
environment. In another approach, execution monitoring is
used to detect failures by checking the consistency of an execution network representing causalities through temporal
SEPTEMBER 2017

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

113
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2017
