IEEE Robotics & Automation Magazine - September 2015 - 101

Planner Arena site: 1) overall performance plots, 2) progress
plots, and 3) regression plots. We will describe these plots in
more detail below.
Plots of Overall Performance
The overall performance plots can show how different planners compare on various measures. The most common performance measure is the time needed for a planner to find a
feasible solution. By default, integer-and real-valued performance metrics (such as solution time) are plotted as box
plots that provide useful summary statistics for each planner:
median, confidence intervals, and outliers. However, in some
cases visualizing the cumulative distribution function can reveal additional useful information. For instance, from Figure 3(a) one can easily read off the probability that a given
planner can solve a particular benchmark within a specified
amount of time. For very hard problems, where most planners time out without finding a solution, it might be informative to look at solution difference: the gap between the
best found solution and the goal [Figure 3(b)]. For optimizing planners, it is often more interesting to look at the best
solution found within some time limit. The overall performance page allows one to select a motion planning problem
that was benchmarked, a particular benchmark attribute to
plot, the OMPL version (in case the database contains data
for multiple versions), and the planners to compare.
Most of the measures are plotted as box plots. Missing data
are ignored. It is important to keep in mind that if a planner
failed to solve a problem 99 times out of a 100 runs, then the
average solution length is determined by one run. To make
missing data more apparent, a table below the plot shows how
many data points there were for each planner and how many
of those were missing values.
Performance is often hard to judge by one metric alone.
Depending on the application, a combination of metrics is
often necessary to be able to choose an appropriate planner.
For example, in our experience LBKPIECE [11] (one of the
planning algorithms in OMPL) tends to be among the fastest planners, but it also tends to produce longer paths. For
time-critical applications this may be acceptable, but for applications that place greater importance on short paths another planner might be more appropriate. There will also be
exceptions to general trends. Bidirectional planners (such as
RRT-Connect [12]) tend to be faster than unidirectional
planners (such as RRT [12]), but Figure 3(a) shows that this
is not always the case. This underscores the need for a good
set of benchmark problems that are representative of different applications.
Progress Plots
Some planners in OMPL are not limited to reporting information after a run is completed, but can also periodically
report information during a run. In particular, for asymptotically optimal planners it is interesting to look at the convergence rate of the best path cost (e.g., path length). By default,
Planner Arena will plot the smoothed mean as well as a 95%

Goal

Start
(a)

Intermediate
Configuration Along
Shortest Path

Goal

Decoy
Homotopy
Class

Start

Tree Produced by Planner
(b)
Figure 4. Two of the sample benchmark problems included on
Planner Arena: (a) with a long, twisty narrow passage and (b) with
several suboptimal decoy homotopy classes of paths.

confidence interval for the mean [Figure 3(c)]. Optionally,
individual measurements can be shown as semitransparent
dots, which can be useful to get a better idea of the overall
distribution. Analogous to the performance plots, missing
data are ignored. During the first couple seconds of a run, a
planner may never find a solution path. Below the progress
plot, we plot the number
of data points available
For community-wide
for a particular planner at
each 1-s time interval.

adoption of benchmarks

Regression Plots
it is important to adopt
Regression plots show
how the performance of
standard input and output
the same planners
change over different
file formats.
versions of OMPL [Figure 3(d)]. This is mostly
a tool for developers
using OMPL that can help in the identification of changes
with unintended side effects on performance. However, it
also allows a user to easily compare the performance of a
user's modifications to the planners in OMPL with the latest official release. In regression plots, the results are shown
as a bar plot with error bars.
Any of the plots can be downloaded as a PDF file or as
RData. The PDF format is useful if the plot is just needs
touch ups. The RData file contains both the plot as well as all
September 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

101

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015