IEEE Robotics & Automation Magazine - September 2015 - 99

further divided into two subcategories: 1) planners that terminate when any solution is found and 2) planners that attempt to compute an optimized solution (with respect to a
user-specified optimization objective). For optimizing planners, a threshold on optimality can be set to control how
close to optimal the solution needs to be. At one extreme,
when this threshold is set to zero, planners will run until
time runs out. At the other extreme, when the threshold is
set to infinity, planners act like nonoptimizing planners and
will terminate as soon as any solution is found.
Typically, a user specifies multiple planners. By default,
OMPL will try to make reasonable parameter choices for each
planner. However, a user can also fine-tune any parameter setting for a planner. With the command line tool's configuration
files, this is easily accomplished by adding lines of the form
planner.parameter=value. The parameter code infrastructure is generic, and when a programmer specifies a parameter for a planner it can be specified through the
configuration file without having to change the parsing of configuration files. It is also possible to add many instances of the
same type of planner. This is useful for parameter sweeps.
Each instance can be given a slightly different name to help
distinguish the results for each instance. Each run of a planner
is executed in a separate thread, therefore, if a planner hangs,
the benchmark program can detect that and forcibly terminate
the planner thread (the run is recorded as a crash and the
benchmarking will continue with the next run).
Database of Benchmark Runs
After a benchmark run is completed, a log file is written out.
With the help of a script, the benchmark results stored in the
log file can be added to a SQLite3 database. Multiple benchmark log files can be added to the same database. The
SQLite3 database facilitates distribution of all relevant benchmark data and users can simply transfer one single file. Furthermore, the database can be easily programmatically
queried with almost any programming language. In contrast,
extracting information directly from the log files or some
other custom storage format would require more effort to
perform the types of analysis and visualization that is enabled by our database schema described below.
Figure 2 shows the database schema that is used. Each
benchmark log file corresponds to one experiment. The experiments table contains an entry for each experiment that
contains the basic benchmark parameters and the detailed information about the hardware on which the experiment was
performed (in the cpuinfo column). Information about each
of the planner instances that were specified is stored in the
PlannerConfigs table. For each planner instance, all parameter values are stored as a string representation of a list of keyvalue pairs (in the settings column). While we could have
created a separate column in the PlannerConfigs table for
each parameter, the parameters are planner specific with very
few shared parameters among planners.
The main results are stored in the Runs table. Each entry
in this table corresponds to one run of a particular planner

trying to solve a particular motion planning problem. After a
run is completed, several attributes are collected such as the
number of generated states (graph_states), duration of
the run (time), length of the solution path (solution_
length), clearance along the solution path (solution_
clearance), and so on. Default solutions are simplified
(through a combination of shortcutting and smoothing
[10]), which usually improves the solution quality at a minimal time cost. Runs can terminate for a variety of reasons,
such as a solution was found, the planner timed out (without
any solution or with an approximate solution), or the planner
crashed. We use an enumerate type for this attribute (stored
in status), and the labels for each value are stored in the
enums table (not shown
in Figure 2).
The infrastructure is aimed
The progress table
stores information periboth at end users who
odically collected during a
run. This collection is
want to select a motion
done in a separate thread
so as to minimize the efplanning algorithm
fect on the run itself.
Progress information is
that performs best on
currently only available
for optimizing planners. It
problems of interest, as
is used to store the cost of
the solution found at a
well as motion planning
particular time. By aggregating progress informaresearchers who want to
tion from many runs for
each planner, we can
compare the performance
compare rates of convergence to optimality (see
of a new algorithm relative
"Interactive Analysis of
Results" section).
to other state-of-the-art
The database schema
has been designed with
algorithms.
extensibility in mind.
Large parts of the schema
are optional and other
columns can be easily added. This does not require new
parsers or additional code. Instead, the log files contain
enough structure to allow planners to define their own run
and progress properties. Thus, when new log files are added
to a database, new columns are automatically added to runs
and progress. Planners that do not report on certain properties will just store the value "N/A" in the corresponding columns. Additional run properties for a new type of planner
are easily defined by storing key-value pairs in a dictionary of
planner data, which is obtained after each run. Additional
progress properties are defined by adding a function to a list
of callback functions.
Log files have a fairly straightforward plain text format
that is easy to generate and parse. (The complete syntax is
specified at http://ompl.kavrakilab.org/benchmark.html.)
This makes it easy for other motion planning libraries to
September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

http://ompl.kavrakilab.org/benchmark.html

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015