IEEE Robotics & Automation Magazine - September 2013 - 52

and infrared images acquired with a ToF camera (as
described in the next section) are segmented into their
composite surfaces as described in the "Depth Segmentation" section. Leaf-model contours are fitted to the
extracted segments, the validity of the fit and the graspability of the leaf are measured, and the segments are ranked
(see the "Extraction of Grasping Points" section). A target
leaf is selected, and the robot moves the camera to a closer,
fronto-parallel view of it. The validity of the target and the
graspability are then re-evaluated (see the sections "Contour Fitting" and "Graspability"). If the leaf is considered
suitable for sampling on the basis of these criteria, then the
probing tool is placed onto the leaf following a two-step
path (see the "Intermediate Goal Position and Probing
Point" section). If the target is not considered suitable for
probing, another target leaf (from the general view) is
selected, and the procedure is repeated.
3-D Image Acquisition
Depth measurements are acquired with a ToF camera [see Figure 1(a) and (b)]. This type of sensor has the main advantage
of providing registered depth and infrared-intensity images of
a scene at a high frame rate. The ToF cameras use the wellknown ToF principle to compute the depth. The camera emits
modulated infrared light to measure the traveling time
between the known emitted waves and the ones reflected back
by the objects in the scene.
ToF cameras have two main drawbacks: low resolution
[e.g., 200 # 200 pixels for a photonic mixer device (PMD)
CamCube 3.0 camera] and noisy depth measurements due to
systematic and nonsystematic errors [17]. On the one hand,
low resolution can be a big problem for large environment
applications, but it does not have such a negative impact
when the camera is used at close ranges as it is our case. On
the other hand, noisy depth measurements due to nonsystematic errors are amplified by working in such a short
range. Systematic errors are highly reduced by calibration
procedures, and nonsystematic errors can be palliated using
filtering techniques. Here we apply two filters to remove
undesired wrongly estimated point depths and noise: a
jump-edge filter and an averaging filter [18].
Depth Segmentation
In this section, we describe an algorithm that segments
the sparse and noisy depth data measured by the ToF
camera into surface patches to extract task-relevant image
regions, i.e., leaves. We assume that plant leaves are usually represented by a single surface in a 3-D space.
Although this assumption may not be generally valid, we
assume that it holds in most cases. With the many occlusions present in grown plants and the variability of leaves
in terms of size, orientation, and 3-D shape, the application of appearance models directly to the image data with
the purpose of the leaf segmentation would be extremely
challenging, especially since partial shape models might
also have to be utilized.
52

IEEE ROBOTICS & AUTOMATION MAGAZINE

september 2013

Removing the noise and invalid points in the depth data by
using the jump-edge filter provides a sparse depth map. We
segment the data by using the infrared-intensity image of the
depth sensor as an auxiliary image. Unlike depth, which is
measured using the ToF principle, the corresponding infraredintensity image provides complete (dense) information with
little noise. In comparison with color or gray-level images, the
infrared intensity images are more amenable to segmentation,
since plant-type characteristic color textures are not present
here. The segments are then selected and merged on the basis
of the available, potentially sparse depth information.
The algorithm proceeds as follows. First, the infrared-intensity image is segmented with a standard algorithm at different
resolutions. The details can be found in [19]. This is necessary
as we do not know beforehand at which resolution good
regions will appear. Those segments that fit the depth data best,
according to a parametric surface model (see the "Fitting of
Quadratic Surface Models" section), are selected, and a new
segmentation is constructed. This procedure has been
described in detail in [15] and will thus not be repeated here.
From this intermediate segmentation and the respective estimated parametric surfaces, a graph is built, in which the nodes
represent segments, and edges represent the pairwise similarity
of the segments' surfaces, as described in the "Segment Graph"
section. Then, to remove remaining over-segmentations present in the intermediate segmentation, a graph-based merging
(clustering) procedure is employed that allows us to handle the
nonlocal character of surface properties (see the "Segment Dissimilarity" and "Graph-Based Merging of Segments" sections).
An overview of the algorithm is provided in Figure 3.
The method requires currently about . 28 s to segment
an image and to fit surface models using MATLAB and nonoptimized code.
Fitting of Quadratic Surface Models
For modeling the 3-D surfaces of image regions, we use a
quadratic function, which allows us to treat planar, spherical,
and cylindrical shapes. Surfaces with more involved curvatures could also be managed within the same approach but
are not required for the application at hand. Moreover, we use
quadratic functions that allow computing the depth z explicitly for the x-y coordinates in the form of z = f ^x, yh . Thus,
surfaces are described by the five parameters a, b, c, d, and e,
where the depth z can be expressed as a function of x and y
through z = ax 2 + by 2 + cx + dy + e .
For a given segment si, we perform a minimization of the
mean squared distance
E i,model = 1
N

/ (z j - z j,m) 2,
j

(1)

of measured depth points z j, m from the estimated model
depth z j = fi,model ^x j, y jh, where fi,model is the data-model
function and N is the number of measured depth points in the
area of segment. The optimization is performed with a NelderMead simplex search algorithm provided in MATLAB.

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2013