IEEE Robotics & Automation Magazine - September 2015 - 94

0.5

same small misalignment as visual output in the case of laser
reconstruction.
The outlier results shown in Figure 12(b) show that both
algorithms increase in the number of outliers as light increases. In this case, stereo reconstruction shows a 0% on
outliers in the absence of light because is not able to obtain
any points. Although in the visual output laser reconstructions seemed to show a higher number of outliers in terms
of percent, it is compensated to the higher amount of reconstruction points, meaning they can be filtered easily.

Stereo Reconstruction
Laser Reconstruction

Coverage (%)

0.4

0.3

0.2

0.1

100 150 200
Light (Lumens)
(a)

0.2

Stereo Reconstruction
Laser Reconstruction

Outliers (%)

0.15

0.1

0.05

100 150 200
Light (Lumens)

250

(b)
Figure 12. (a) and The reconstruction coverage and (b) outlier
results for a stereo camera and a laser in real scenarios for different
light conditions.

The laser reconstruction and stereo reconstruction have similar
comparison system
mean errors, around
0.008 m. This result is
in simulation and
much greater than in simulation due to the added
controlled environments
ground truth estimation
error. The standard deviahelps to establish an
tion, though, is greater
than the one obtained in
objective benchmarking
simulation, around 0.005
m in stereo and 0.008 m
methodology.
in the case of laser reconstructions, however, it is
small enough to conclude
that the tested algorithms reached a good alignment and the
ground truth estimation was fairly good. This shows the

The use of an automated

IEEE ROBOTICS & AUTOMATION MAGAZINE

September 2015

Conclusions
In this article, a benchmarking process is presented to allow
easy objective comparison and replication of the results of
two 3-D object reconstruction algorithms as a prior step to
manipulation in simulated and real scenarios. This process
involves a benchmarking platform developed to evaluate
software using a simulator as ground truth for the evaluation. The presented results show the potential of the benchmarking techniques to obtain measurable results in
simulated scenarios, just as in real situations, helping to decide which approach is better in each situation so that the
design of the system can be improved at an early stage. Results replicability is assured as the simulator, benchmarking
platform, and the algorithms used to test it are offered as
open source. Furthermore, key parameters, such as light
conditions, have been measured in order to provide sufficient information for the experiment to be replicable. As a
work in progress, an online benchmarking platform is
being actively developed to avoid software installation and
make algorithm evaluation and comparison faster and
more user-friendly.
Acknowledgments
This work was partly supported by Spanish Ministry under
grant DPI2014-57746-C3 (MERBOTS Project); by Universitat Jaume I grants PID2010-12, PREDOC/2012/47, and
PREDOC/2013/46; and by Generalitat Valenciana grant
ACIF/2014/298.
References
[1] (2014, Dec. 3). Survey and inventory of current efforts in comparative robotics research. EURON-European Robotics Search Network. [Online].
Available: http://www.robot.uji.es/EURON/en/index.htm
[2] W. Nowak, A. Zakharov, S. Blumenthal, and E. Prassler. (2010, Apr.).
Benchmarks for mobile manipulation and robust obstacle avoidance and navigation in Deliverable D3.1 from FP7-BRICS Project (Best Practice in Robotics).
[Online]. Available: http://www.best-of-robotics.org/home
[3] R. Bischoff, T. Guhl, E. Prassler, W. Nowak, G. Kraetzschmar, H.
Bruyninckx, P. Soetens, M. Haegele, A. Pott, P. Breedveld, J. Broenink, D. Brugali, and N. Tomatis, "BRICS-Best practice in robotics," in Proc. IFR Int.
Symp. Robotics, June 2010, pp. 968-975.
[4] Specification of benchmarks. (2009, Jan.). in Deliverable D6.1 from FP7DEXMART Project (DEXterous and autonomous dual-arm/hand robotic manipulation with sMART sensory-motor skills: A bridge from natural to
artificial cognition. [Online]. Available: http://www.dexmart.eu

http://www.robot.uji.es/EURON/en/index.htm http://www.best-of-robotics.org/home http://www.dexmart.eu

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015