IEEE Robotics & Automation Magazine - June 2020 - 57

approach to ours using similar CNN architecture. In contrast
to our work, they did not interpret planning as probabilistic
inference; they optimized only for hand pose, ignoring handjoint configurations, and they validated only in simulation.
Veres et al. [22] trained a conditional variational autoencoder deep network to predict the contact locations and
normals for a multifingered grasp given an RGB-D image of
an object. To perform grasping, an external inverse kinematics solver must be used for the hand to try to reach the
desired contact poses as best as possible. Liu et al. [8]
trained a 3D voxel CNN to directly predict the multifinger
grasp configuration. Implicit in such regression methods, as
proposed in Liu et al. [8] and in Veres et al. [22], is the
assumption that there is a unique best grasp for a given
object view. In contrast, our method can plan multiple successful grasps for a given object using different initial configurations with associated high confidence prior to
execution. This offers the robot the option of selecting a
grasp best suited for its current task. Additionally, we show
that our classification-based network can effectively learn
with a smaller data set compared to a regression network,
which cannot leverage negative grasp examples.
We formulated multifingered grasp planning as probabilistic inference in a learned DNN without a prior over grasp
configuration in our previous work [24]. Our multichannel
DNN in Lu et al. [24] took a grasp configuration and an
RGB-D image grasp patch as inputs and predicted as output
the probability of grasp success. Our planning algorithm generally achieved higher grasp-success rates compared with
sampling-based and regression approaches currently used for
grasping with NNs.
We explored a probabilistic graphical model for grasp
learning and planning over grasp type and configuration for a
given object in our previous work [26]. We used a data-driven
Gaussian mixture model (GMM) prior independent of the
object to constrain the inference not to stray into areas far
from grasp configurations observed at training time, where we
have little evidence to support grasp-success predictions. The
grasping experiment results in Lu and Hermans [26] demonstrated the benefit of a data-driven prior for grasp inference. In
this article, we propose an object-conditional prior modeled as
an MDN [27]. Our MDN prior models the grasp-configuration distribution based on the geometry of the object of interest. Our real-robot experiments show that the grasp inference
with the MDN object-conditional prior outperforms grasp
inference with the GMM object-independent prior. We
trained a logistic regression classifier to predict the grasp-success probability on a small data set with 120 grasps in Lu and
Hermans [26]. In this article, we trained a voxel-based 3D
CNN to predict the grasp-success probability on a larger data
set containing 10,811 grasp attempts.
Grasp Planning as Probabilistic Inference
Following Ciocarlie et al. [11], we defined the grasp-planning
problem as finding a grasp preshape configuration. In our
case, the grasp-configuration vector is composed of the palm

pose in the object-reference frame and the hand's preshape
joint angles, which define the shape of the hand prior to closing
the hand. To make the grasp inference agnostic to object poses,
we put the palm pose in the object-reference frame for learning
and inference. After finding the grasp preshape configuration,
the robot moved to this preshape and ran a controller to close
the hand forming the grasp on the object. We explain the specific joints used for defining the preshape and how the grasp
controller works for our experiments in the section "Grasp
Data Collection." We focus on scenarios where a single, isolated
object of interest is present in the scene. Importantly, we assume
no explicit knowledge of the object beyond a single camera sensor reading of it in its current pose. The problem we address is
stated as follows: given such a grasp scenario, plan a grasp preshape configuration that allows the robot to successfully grasp
and lift the object without dropping it.
Given the learned model parameters, W and U, along
with the visual representation, z, associated with an observed
object of interest, our goal is to infer the grasp-configuration
parameters, i, that maximize the posterior probability of
grasp success Y = 1. Here Y defines a random Boolean variable, with 0 meaning failure and 1 meaning success. We can
thus formalize grasp planning as a maximum a posteriori
(MAP) inference problem:
argmin - log p ^i Y = 1, z, W, Uh
i

subject to i min ) i ) i max .

(1)
(2)

We constrain the grasp-configuration parameters to obey the
joint limits of the robot hand in (2).
We define the grasp-success likelihood p (Y = 1 | i, z, W)
to be a DNN. W represents the NN parameters. The DNN
predicts the probability of grasp success, Y, as a function of
the visual representation of the object of interest, z, and hand
configuration, i. (We previously represented the object of
interest as an RGB-D image patch in Lu et al. [24]. In this
article, we use a voxel grid to represent the object of interest.)
We describe the details of our NN classifier in the section
"Voxel-Based Grasp-Likelihood Classification."
We present three different ways to model the prior over
the grasp configuration i. In each case, U represents the
associated parameters of the prior distribution. In the first
approach, we assume a uniform prior over valid grasp configurations, resulting in the grasp-success posterior probability
being proportional to the likelihood:
p ^i Y = 1, z, W, Uh ? p ^Y = 1

i , z, W h .

(3)

This prior requires all grasp parameters to be bounded to prevent the inference straying from the training evidence. This
approach was used in our initial work [24]. It is trivial to
bound the preshape joint angles using the robot hand joint
limits. However, it requires heuristic bounds to be manually
designed for the hand palm pose in Cartesian space.
The second prior we examine defines a prior over grasp
configurations to encode preferred grasp configurations
JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020