IEEE Robotics & Automation Magazine - June 2020 - 62

performed 240 grasp attempts for three different methods
across eight objects in this article.
Our evaluation protocol on the physical robot mirrors that
used in simulation. Namely, we labeled a grasp attempt that
lifted the object to a height of 0.15 m without dropping it as
successful. We used the same motion planner and grasp controller that we used in data collection to plan paths for the
arm and close the hand. If the motion planner failed to generate a plan for a grasp due to either inverse kinematics or collision avoidance, we generated a new grasp using the same
grasp planner with a different initialization. If the grasp planner could not generate a grasp with a motion plan in five
attempts, we treated the grasp attempt as a failure.
The MDN object-conditional and GMM priors both have
two mixture components. For both learned prior models,
we found the grasp-configuration mean of one mixture

80
60
40
20
All

Soccer Ball

Sugar Box

Soft Scrub

Mug

Mustard

Lego

Pitcher

0
Pringles

Success Rate (%)

100

Voxel-MDN-Prior
Voxel-GMM-Prior
Voxel-Uniform-Prior
Figure 4. The multifingered side-grasping success rates of three
different prior distribution methods on the real robot. The
Pringles object was seen in training. The other seven objects
were previously unseen.

80
60
40
20
All

Soccer Ball

Sugar Box

Soft Scrub

Mug

Mustard

Lego

Pitcher

0
Pringles

Success Rate (%)

100

Voxel-MDN-Prior
Voxel-GMM-Prior
Voxel-Uniform-Prior
Figure 5. The multifingered overhead-grasping success rates of
three different methods on the real robot. The Pringles object was
seen in training. The other seven objects were previously unseen.

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

component is a side grasp (we term this component the sidegrasp component) and the grasp-configuration mean of the
other component is an overhead grasp (we term this component the overhead-grasp component). We randomly sampled a
grasp configuration from the side-grasp component to initialize
the grasp inference to generate a side grasp. Similarly, we initialized the grasp inference with a random sample from the overhead-grasp component to plan an overhead grasp. We
initialized the grasp inference of the uniform prior with side
and overhead grasps generated by the heuristic geometry-based
planner used for data collection to plan side and overhead
grasps respectively.
The side-grasp success rates for all three methods are summarized in Figure 4. Figure 5 shows the overhead-grasp success rates for all three methods. It takes 5-10 s for each method
to generate a grasp. The grasp planners using the MDN, the
GMM, and the uniform priors achieved success rates of 75, 58,
and 3%, respectively, on side grasps for the eight objects. The
grasp planner was not able to plan any successful side grasps
for the mug object. Since the mug object is relatively short, the
motion planner could not find paths that would not collide
with the table during side grasps for all three methods. The
grasp planners using the MDN, GMM, and uniform priors
achieved success rates of 40, 10, and 0%, respectively, on overhead grasps of the eight objects. The grasp planner never generated overhead grasps successfully for the mustard and Lego
objects. The mustard and Lego objects have relatively smaller
contact areas available for overhead grasps, and our grasp controller would push them away when closing the hand as it had
no feedback from vision or haptic sensors to know that the
object was moving.
The grasp planner using the MDN prior achieved higher
success rates than the two other methods for both side and
overhead grasps; this demonstrates the benefit of modeling
the grasp prior conditionally on the observed object when
performing inference. The uniform prior achieved the lowest
success rates for both side and overhead grasps. This shows
that data-driven priors, even if not conditioned on the
observed object, outperform the use of a heuristic planner for
initialization and locally constraining the grasp configuration,
as these heuristics cannot reliably generate successful grasps.
In Figure 6, we show example grasps for different objects
generated by our inference approach with the voxel-based
classifier and MDN prior. Grasps in the top two rows [(a)-
(n)] are side grasps, which provide strong stability. The bottom two rows [(o)-(bb)] show overhead grasps, which
provide access to objects in clutter and often demonstrate
improved dexterity between the robot and object.
Discussion and Conclusions
In this article, we presented a novel approach for multifingered grasp planning formulated as probabilistic inference
in learned DNNs. We proposed a voxel-based 3D CNN to
predict the probability of grasp success as a function of both
the object voxel grid and grasp configuration. We showed that
our novel, voxel-based grasp-success classifier for

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020