IEEE Robotics & Automation Magazine - September 2018 - 69

Motion domain: The motion domain block shows the definition for a single object (a glass). The definition includes the
object's relative position to its parent (the shelf) and the
object's geometric mesh. The full task domain includes similar definitions for the other glasses and bowls as well as the
links and joints of the robot.
Domain semantics: The domain semantics block shows the
function to find a motion plan for the pick-up action. This
function computes the current position of the object, then
attempts to find a motion plan to bring the robot's hand to a
grasping pose for that object. If motion planning fails
(exceeds a timeout), the motion planning function generates
an exception that the TM planner will catch and handle by
finding a different task plan based on the feedback from the
motion planner.
TM planner: The TM planner block illustrates the alternation and feedback between task planning and motion planning. The task planner identifies a high-level plan. The
motion planner attempts to find corresponding paths. Failing
to do so, the motion planner provides additional constraints
to the task planner, which then finds a different task plan.
This process iterates until it finds a task plan where all actions
have corresponding motion plans.
TM plan: The TM plan block shows the first two actions of
the plan: picking and placing an object. The first action
(pick-up) includes the joint waypoints to move the robot's
hand to the grasping position for an object, then changes the
object's parent in the scene graph to the robot's hand. The second action (put-down) includes the waypoints to move the
robot's hand and the grasped object to the desired location,
and then (unshown) the object's parent will change in the
scene graph to the table, placing the object. The full TM plan
contains the rest of the actions necessary to achieve the
desired goal.
Plan execution: Figures 7 and 8 show two TM plans and
planning times, one for the Rethink Robotics Baxter and one

for the Universal Robots UR5. The same overall framework
produces the plan for each system. We apply the framework
in each case by using the URDF model of the robot for the
specific system.
These examples demonstrate the modularity and extensibility of TMKit. TMKit works on multiple robots, supports
multiple types of actions (e.g., picking, placing, stacking, and
pushing), and handles coupling between objects [e.g., moving
cans into a bin (Figure 8)]. Additional benchmark results are
presented in [1] and [2].
Conclusions
We have presented TMKit, a new software framework for
TMP and execution that is available under an open-source,
permissive license [21]. We believe TMKit is the first opensource TMP framework that is extensible to multiple domains
and different planning methods and that supports end-to-end
planning and execution. Its modular design enables TMKit to
generalize across hardware platforms, task domains, and
TMP algorithms.
We produced a general-purpose, easy-to-use, and extensible framework for TMP. There are numerous avenues to
improve and build upon this framework. Going beyond our
implemented method of [1] and [2], we will extend the feedback between the task and motion layers, improve plan reuse,
and incorporate additional rich constraint capabilities. We are
already adapting and integrating the additional TMP methods of [3] and [4] with TMKit. We hope the community will
find this end-to-end system both easy to use and a helpful
platform to demonstrate other methods for TMP.
Currently, TMKit focuses on the geometric case of motion
planning, which is often sufficient for manipulation. Planning
with dynamics, e.g., considering torques in the planning layer,
may be necessary for other cases, such as bipedal walking.
However, including dynamics during motion planning may
impact completeness [20], so careful analysis is necessary.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 8. A TM plan to load and move a bin using the Universal Robots UR5. The same overall framework with a different URDF
for the robot produces this plan for a different system. The average planning time for ten trials was 8.78 s on an Intel Core i7-4790.
(a)-(f) The simulated execution and (g)-(l) the physical execution.

september 2018

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

69
IEEE Robotics & Automation Magazine - September 2018

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2018
