IEEE Robotics & Automation Magazine - September 2015 - 97

Planning Library (OMPL, http://ompl.kavrakilab.org), a C++
library that contains implementations of many sampling-based
algorithms [3]. One can immediately compare any new planning algorithm to the more than 30 other planning algorithms
that currently exist within OMPL. There is also flexibility in
the types of motion planning problems that can be benchmarked, as discussed in the "Defining Motion Planning Problems" section. Second, we have defined extensible formats for
storing benchmark results. The formats are fairly straightforward so that other planning libraries could easily produce
compatible output. Finally, we have created an interactive, versatile visualization tool for compact presentation of collected
benchmark data (see http://plannerarena.org). The tool and
underlying database facilitate the analysis of performance
across benchmark problems and planners. While the three
components described above emphasize generality, we have
also created a simple command line tool, specifically for rigid
body motion planning that takes as input a plain text description of a motion planning problem.
Benchmarking sampling-based planners is nontrivial for
several reasons. Since these planners rely on sampling, performance cannot be judged from a single run. Instead, benchmarks need to be run repeatedly to obtain a distribution of
some performance metric of interest. Simply comparing the
means of such distributions may not always be the correct
way to assess the performance. Second, it is well known that
different sampling strategies employed by sampling-based
algorithms typically perform well only for certain classes of
problems, but it is difficult to exactly define such classes.
Finally, different applications require optimization for different metrics (e.g., path quality versus time of computation)
and there is no universal metric to assess performance of
planning algorithms across all benchmarks.
There have been some attempts in the past to come up
with a general infrastructure for comparing different planning algorithms (see [4], [5]). This article is in the same spirit
but includes an extended and extensible set of metrics and
offers higher levels of abstraction and concrete entry-level
points for end users. Furthermore, we also introduce an extensible logging format that other software can use and a visualization tool. To the best of our knowledge, none of the
prior work offered the ability to interactively explore and visualize benchmark results. The Motion Planning Kernel
(MPK) software system described in [4] is similar to OMPL
in that both aim to provide a generic, extensible motion
planning library, but MPK appears to no longer be maintained or developed. There has been significant work on
metrics used for comparing different planning algorithms
(see [6], [7]), and our benchmarking infrastructure includes
many of these metrics.
The contribution of this article is not to any particular
benchmark problem, metric, or planner but provides a generic, extensible benchmarking infrastructure that facilitates easy
analysis and visualization of replicable benchmark results.
Since it is integrated with the widely used and actively developed OMPL, it becomes straightforward to compare any new

Benchmark
Database of
Configuration Files Benchmark Problems

OMPL

Log Files

Other Planning
Libraries

MoveIt!

Database of
Benchmark Results

Interactive
Visualization

Figure 1. An overview of the benchmarking infrastructure.

motion planning algorithm to many other state-of-the-art
motion planning algorithms. All relevant information pertaining to how benchmarks were run is stored in a database to
enable replicability of results.
Benchmarking Infrastructure
OMPL provides a high level of abstraction for defining motion planning problems.
The planning algorithms
There is still no
in OMPL are, to a large
extent, agnostic with recharacterization of which
spect to the space they are
planning in. Similarly, the
algorithms are well-suited
benchmarking infrastructure within OMPL allows
for which classes
the user to collect various
statistics for different
of problems.
types of motion planning
problems. The basic
workflow is as follows.
● The user defines a motion planning problem. This involves
defining the state space of the robot, a function that determines which states are valid (e.g., collision-free), the starting
state of the robot, and the goal. The complete definition of a
motion planning problem is contained within a C++ object, which is used to construct a benchmark object.
● The user specifies which planning algorithms should be
used to solve the problem, time, and memory limits for
each run and the number of runs for each planner.
● The benchmark is run. Upon completion, the collected results are saved to a log file. A script is used to add the results in the log file to a SQL database. The results can be
queried directly in the database or explored and visualized
interactively through a website set up for this purpose
(http://plannerarena.org).
September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

http://ompl.kavrakilab.org http://www.plannerarena.org http://www.plannerarena.org

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015