IEEE Robotics & Automation Magazine - December 2023 - 15

side branch detection to a leader (if possible) and determine
the location of intersection. The method is illustrated in Figure
6. First, all of the detected masks are decomposed using
principal component analysis to yield a primary orientation
and center. Each leader branch is assigned a width w equal to
the average number of pixels in the horizontal direction along
the mask. Each side branch-leader pair is checked to see if
their point of intersection p*
any part of the side branch mask is sufficiently close to the
leader boundary (at width w). If so, letting bv represent the unit
vector for the side branch (facing away from the trunk), the
pixel pruning point is computed as
p `jv where
w mb
2
*++ ,
m 90= is a manually specified pixel offset from the join
point between the leader and the side branch. This process is
repeated for every side branch-leader pair.
Finally, each 2D pixel estimate must be converted into a 3D
location to move the robot. To convert the pixel estimate into
a 3D location, we make an assumption that the pixel is located
on a plane 30 cm away from the camera's optical frame in the
z-direction. The system uses the intrinsics of the camera to
yield a 3D estimate of the point in the camera's optical frame,
which is then transformed using the kinematic model of the
arm to obtain a world frame estimate for the pruning point.
This estimate of the pruning point in the world is not particularly
precise since the 30-cm planar estimates for the pruning
points will not always be true, and the 90-pixel offset is chosen
arbitrarily. However, the " Closed-Loop Approach (Waypoint
to Cutpoint) " section discusses the approach phase controller
that corrects for this error.
Once the system identifies the pruning points,
it must
choose an order in which to cut them. We have explored task
sequencing for pruning in our previous work [27]. However,
for this pruning setup, the close-up view of the scene means
there are rarely more than two eligible targets to prune. Therefore,
we choose to sequence the points arbitrarily.
CLOSED-LOOP APPROACH (WAYPOINT TO CUTPOINT)
Once a pruning point is detected and its position determined,
the final step of the pruning process is to move the
pruning implement to the target point and execute a cut. If
this pruning point estimate is perfectly accurate, then the
system simply needs to plan a path that does not collide
with any obstacles to move the cutters and cut. This openloop
method of planning, in which the system does not use
sensor feedback as it moves the manipulator toward the
goal, has been the dominant approach for previous pruning
systems. However, many external factors (e.g., vehicle slippage,
sensor noise, and wind) can affect the accuracy of the
original estimates. Furthermore, such open-loop approaches
have no way of regulating the dynamic interaction of the
manipulator with the environment, which can lead to damage
to the environment or to the robot. These issues motivated
the development of a hybrid controller for the
approach phase that uses both visual and force feedback for
accurate cutting, the first of its kind to be used in an endto-end
pruning system.
The details of the hybrid controller are explained in our
lies inside of the leader and if
previous work [5]. The visual controller is a deep neural
network that takes in the segmented version of the scene,
as described in the " Branch Segmentation " section, i.e., a
two-channel image corresponding to masks for the foreground
branches and the cutter as well as the colorized
optical flow image to yield a five-channel input. It outputs
a control action (, )[ ,] [, ].
vv ! -- Given a forxy
11
11
#
ward velocity of s .,= 003 the end effector is commanded
to move at a Cartesian velocity of [, ,]
sv sv sxy in the end
effector's frame. We chose to run the visual controller at
one frame per second to obtain reasonable optical flow
Leader
Side Branch
Spur
Other Branch
Nonbranch
FIGURE 5. An example of a labeled image, showing five labeled
classes of foreground objects: leaders, side branches, small
spurs, " other " branches (usually offscreen branches that do not
definitively fit in one of the former categories), and nonbranch
objects.
Leader Detection
Side Branch Detection
Pruning Point
Join Point
Width
Side Branch
Attachment
Margin
FIGURE 6. The process of matching a detected side branch with
a leader. Each detection is represented as a line segment. If a
side branch intersects the leader line segment and the mask is
sufficiently close to the leader, the pruning point is computed a
fixed offset away from the estimated join point between the side
branch and leader.
DECEMBER 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
15
IEEE Robotics & Automation Magazine - December 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2023
