IEEE Robotics & Automation Magazine - September 2015 - 93

cases, higher Gaussian noise means a higher number of
outliers. In the case of stereo vision, noise and the absence
of light makes it more difficult to match the pixels on both
cameras, and a large number of outliers appear. On the
other hand, in strong light conditions, it is more difficult
to find the laser light on the camera and mistaken detections cause a high number of outliers. It is also remarkable
that there is a nonnegligible number of outliers in the stereo vision results for 0.00% standard deviation noise.
These outliers are caused by small floating particles simulated in UWSim, which are correctly detected by the system although they are not part of the object. The 3-D
stereo reconstruction is not able to find these outliers
when the noise is higher and its impact decreases as more
parts of the object are correctly reconstructed.
Real Results
In the experiment conducted under real conditions, both systems have been tested under three
different light conditions, shown in Figure 10, in
order to replicate the simulation results. As the illumination of the environment is a key characteristic in this experiment, a lux meter was used to
assure the replicability of the experiment. The lux
meter was placed in a flat surface close to the
black box. The values obtained for the testing scenarios were 12, 147, and 207 lumens.
The system has been calibrated in such a way
so that the benchmarking module for UWSim
can be used to measure the real results in the
same way as it is used to measure simulated results. In order to do this, the input data must be
configured to be taken from the real world instead of a virtual scene and use the calibration
method mentioned above in order to acquire the
ground truth.
The 3-D point clouds reconstructed are then
evaluated by the benchmarking platform, as can
be seen in Figure 11, where real point clouds
are displayed on UWSim while being processed. In the images, black dots are filtered as
ground points, green points are considered object points, and blue points are labeled as outliers. Similar to the previous simulation, laser
reconstruction works better in low-light environments, while stereo reconstruction needs
some light to work properly.
A further analysis of the visual results shows
that although laser reconstructions look better,
there is a misalignment on the point cloud. The
3-D laser reconstructions are slightly rotated with
respect to the ground truth target due to small errors in the camera to laser projector calibration.
In the case of stereo reconstruction, the ground
reconstruction was very poor due to the absence
of texture on it.

Results for coverage are shown in Figure 12(a). Laser
results are slightly worse than simulated ones. In this case,
laser achieves around
32-38%, while in simulation it reached 50%.
An open source
Although laser works
better in dark situations,
benchmarking module
it is highly resistant to
light changes. On the
is available for UWSim
other hand, the stereo
reconstruction is comthrough ROS interfaces.
pletely dependent on
light conditions, achieving a 38%, of coverage in
good light environments. These results support the ones
obtained in simulation where both algorithms behave in a
similar way.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 10. (a) and (d) The low, (b) and (e) medium, and (c) and (f) high light
conditions. (a)-(c) External view. (d)-(f) Camera view.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 11. (a) and (d) The low, (b) and (e) medium, and (c) and (f) high light
conditions. (a)-(c) Real laser point cloud reconstruction overlayed on the
UWSim. (d)-(f) Real stereo point cloud reconstruction overlayed on the UWSim.

September 2015

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

93
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2015
