IEEE Robotics & Automation Magazine - September 2013 - 51

related but new research topic in agricultural robotics with
many potential applications. For example, probes could be
taken from plants automatically to detect plant diseases or
nutritional deficiencies. The treatment of singular plants can
then prevent spreading of disease in fields and reduce the
application of chemicals. Another potential application is
the fast probing of plants in research laboratories for
phenotyping purposes. We expect to face similar challenges
as the ones previously encountered with picking robots in
agriculture:
1) the recognition and localization of the target, e.g., fruits
and leaves, given the varying appearances of plants
2) the probing, grasping, cutting, or detachment of parts of
the plant under weakly constrained conditions in natural
environments.
Another major challenge in agricultural robotics is the
guidance of motions through crop fields or greenhouses,
which is not addressed in this article. The first challenge
requires new solutions for the recognition and localization
of leaves to be developed. Previously, color vision has been
used to obtain some relevant plant features, mainly for recognition and classification purposes [7], but when it comes
to extracting structural/geometric information for 3-D
modeling and robot manipulation, the concourse of a user
is required to provide hints on the segmentation from multiple views [8]. If a fully automated process is sought, depth
information needs to be extracted through stereo [9], structured light [10], or a laser scanner [11]. These techniques
have been proven adequate for offline modeling, but they
either require special conditions or are too slow to be used
in online robot interaction with plants. Recently, time-offlight (ToF) cameras have been proposed as a good alternative [12] since they provide low-resolution depth images at
25 frames/s. This permits quick acquiring and fusing of the
images from different viewpoints [13], which is very useful
since one-shot plant data are often partial or ambiguous.
Concerning the robot's actions, planning and learning
algorithms for the manipulation of deformable objects [14]
play an important role in this context. Planning must

encompass the motion of the camera as well since plants are
prone to occlusions and merging of close leaves; selecting
the best next viewpoint may be crucial to disentangle
occluded leaves [15] as well as to determine and access suitable probing points.
Specifically, we address the problem of accurately placing
a cutting tool on a leaf to acquire sample discs from plants.
Samples drawn at different developmental stages can be
used to subsequently analyze their relative growth rates [16].
Thus, the emphasis of this article is on sensing-for-action
methods developed to segment leaves, fit quadratic surfaces
to them, determine best candidates for probing, move the
cameras to get a closer view, determine a suitable sampling
point on the chosen leaf, and finally reach this point with a
disc-cutting tool. Intensity-based segmentation is complemented with the depth data supplied by a ToF camera to
delimit and fit surface patches to the leaves. The ToF camera
and the cutting tool are mounted on the robot end-effector
(as shown in Figure 1) so that an egocentric coordinate
frame is used for all motions.
Overview of the Method
The probing of a leaf follows a two-stage approach (see Figure 2). Initially, the robot arm is moved to a position from
which a general view of the plant is obtained. The depth

Move to Initial Far Position and
Acquire Depth/IR Images

Extract and Evaluate
Potential Target Leaves

Select Target
Leaf

Move Robot to Close,
Frontal View of Target Leaf and
Acquire Depth/IR Images

Extract Target Leaf and
Grasping Points
(b)

Suitable?
(a)

(c)

Figure 1. (a) The WAM arm used in the experiments holding the
ToF sensor, a color camera (data not used), and the cutting tool
used to extract samples from the leaves. (b) Typical intensity
image. (c) The color-coded 3-D point cloud acquired with a ToF
camera [200 # 200 photonic mixer device (PMD) CamCube 3.0].

Yes
Sample Leaf in
Two-Step Path

Figure 2. A flow chart of the suggested probing procedure.

september 2013

IEEE ROBOTICS & AUTOMATION MAGAZINE

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2013