IEEE Robotics & Automation Magazine - June 2020 - 56

performing multifinger grasping on a physical robot. Our
experimental results show that our planning method outperforms existing grasp-planning methods for neural networks (NNs).
Overview and Motivation
Learning-based approaches to grasping [1]-[8] have become
a popular alternative to geometric [9]-[12] and model-based
planning [13], [14] over the past decade. This is largely
because grasp learning generalizes well to previously unseen
objects where only partial-view visual information is available. Moreover, by having a robot attempt and validate its
own grasps in a self-supervised manner [4], we remove the
need for humans to make guesses about what grasps will be
successful [1] as well as the need for the performance of
complex mappings from human grasps to robot grasps.
Recently, researchers have looked to capitalize on the success of DNNs to improve grasp learning. Broadly speaking,
DNN methods for grasp learning can be split into two
approaches: 1) predicting grasp success for the object visual
information represented by an image patch or a pointcloud
associated with a gripper configuration [3], [4], [6], [15]-[19]
and 2) directly predicting a grasp configuration from the
object visual information represented by an image or image
patch using regression [8], [20]-[22]. While these deeplearning approaches have shown impressive performance for
parallel jaw grippers (e.g., [4]), little work has focused on the
more difficult problem of multifingered grasping [7], [8],
[19], [22], [23]. We believe two primary difficulties restrict
the use of deep learning for multifingered grasping:
1) the grasping input representation in NNs
2) the reliance on external planners for generating candidate
grasps.
The first difficulty arises from
1) the great variability in object shapes a robot may encounter
during deployment
2) the high dimensionality of multifingered grasp configurations.
As such, grasp learning requires representations that can
efficiently, in terms of data and computation, encode both the
geometry of the object and the grasp configuration necessary
to predict grasp success. The second issue arises from the
increase in search complexity for planning multifingered
grasps compared to grasping with parallel jaw grippers. As
such, we desire an efficient inference procedure that can
accomplish tasks faster than typical model-based planners
and that does not suffer from the limiting assumptions and
bias present in such human-designed planners.
To combat these two problems, we propose an alternative
approach to grasp planning with DNNs. With this approach,
we directly use the learned network for planning. In our
work, we trained a network to predict grasp success, given an
object and grasp-configuration representation. However,
unlike currently employed sampling methods, we performed
a continuous optimization over the grasp configuration, guiding the updates with the network.
56

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

In addition to giving the NN the object visual information
represented by a red, green, blue, and depth (RGB-D) image
or a voxel grid as input, we provided the grasp-configuration
parameters in the form of finger-preshape joint angles and
palm pose. However, we formulated the learning and inference problem in a general form that would allow other grasp
representations to easily be used instead (e.g., joint angles of
in-contact fingers, desired finger contact locations on the
object surface, and others).
Once trained, given the object visual representation, we
performed inference over the grasp-configuration parameters
to maximize the probability of grasp success learned by our
CNN. We performed this probabilistic inference as a direct
optimization over the grasp configuration, which leverages
the efficient computation of gradients in NNs, while ensuring
joint angles remain within their limits. Thus, our approach
can quickly plan reliable multifingered grasps given an image
of an object and an initial grasp configuration.
This article makes the following contributions over our
previous work [24]:
●●
We propose a new grasp model for grasp learning and
inference, including a voxel-based 3D CNN for grasp-success probability prediction that encodes the object 3D
geometry well.
●●
We present a novel MDN to model a grasp-configuration
prior conditioned on the observed object, which removes
the need for an external grasp planner to initialize our
grasp optimization.
●● Our grasp planner generates both successful side and overhead grasps on a real robot, while our grasp planner,
described in Lu et al. [24], only generates successful side
grasps on the real robot.
●● We train our grasp model using more than seven times the
data of Lu et al. [24].
●● We perform real-robot grasp experiments on more objects
than in Lu et al. [24].
●●
In total, this creates a grasp planner that achieves a higher
success rate than our previous planner, described in Lu
et al. [24].
Our planner offers a number of benefits over previous
deep-learning approaches to multifingered grasping. Kappler
and colleagues [23] learned to predict whether a given palm
pose will be successful for multifingered grasps using a fixed
preshape and performed planning by evaluating a number of
sampled grasp poses. Varley et al. [19] presented a deeplearning approach to effectively predict a grasp-quality metric
for multifingered grasps, but they relied on an external grasp
planner to provide candidate grasps. In contrast, our method
learns to predict grasp success as a function of both the palm
location and preshape configuration and plans grasps directly
using the learned network. Saxena et al. [2] also performed
grasp planning as inference using learned probabilistic models; however, they used separate classifiers for both the image
and range data, and they used hand-selected models instead
of a unified deep model. Zhou and Hauser [25] concurrently
proposed a similar optimization-based grasp-planning

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020