IEEE Robotics & Automation Magazine - June 2020 - 58

independent of the observed object, which gives the following posterior:
p ^ i Y = 1 , z, W, U h ? p ^ Y = 1

i , z, W h p ^ i

Uh.

(4)

In practice. one could choose from many different functions
to implement this prior; however, in this article, we focus on
using a GMM, building on our previous use of GMM grasp
priors in Lu and Hermans [26].
As a third prior, we propose an object-conditional prior:
p ^i Y = 1, z, W, Uh ? p ^Y = 1 i, z, W h p ^i z, Uh. (5)
Here, the preferred grasp configurations are conditioned on
the observed sensory information. For both the GMM of (4)
and the object-conditional prior of (5), we examine data-driven priors to encode knowledge of what data were observed by
the learner during training. This prior thus prefers grasps
similar to those seen during training. Such priors can be
viewed as an approximation of the epistemic uncertainty of
the learned classifier [28], encoding the belief that the classifier's confidence should decrease for grasp or objects far from
those observed during training. We define the details of the
object-conditional prior as a DNN in the section "VoxelBased Grasp Prior Networks." In the section "Robotic Grasp
Inference Experiments," we describe experiments we performed comparing grasp inference using variants of these
three priors.
We solved the inference problem for all three grasp models
in the log-probability space and regularized the log-prior with
a multiplicative gain of 0.5 to prevent the prior from dominating the inference. We used the popular limited memory Broyden-Fletcher-Goldfarb-Shanno optimization algorithm with
bound constraints [29], [30] to efficiently solve the inference
problem. We used the scikit-learn library (http://scikit-learn
.org/stable/index.html) to perform the optimization. We initialized the inference by randomly sampling from the learned
priors. We initialized the uniform prior using a heuristic
described in the section "Grasp Data Collection" as previously
done [24].
Voxel-Based Deep Networks for Multifingered
Grasp Learning
Voxel-Based Grasp-Likelihood Classification
Our voxel-based neural network classifier predicts grasp success for multifingered hands. Figure 1(a) shows the architecture of our grasp-success prediction network. Our
voxel-based classifier takes three inputs: a 32 # 32 # 32
object voxel grid; a vector defining the width, height, and
depth of the voxel grid in the 3D scene; and a 14D vector
encoding the grasp preshape configuration, which we define
in more detail in the section "Grasp Data Collection." This
preshape configuration could be replaced with other grasp
representations in a straightforward manner. The network
processes the object voxel grid with a subnetwork composed
58

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

of four 3D convolutional layers and one fully connected layer,
which we named the voxel encoder. We pretrain this voxel
encoder on a 3D object-reconstruction task described in the
section "Grasp Model Training."
We concatenate the voxel features processed by the voxel
encoder with the object-size vector, and we pass the concatenated features through two fully connected layers to generate
the final object-feature representation. The grasp-configuration input is processed by two fully connected layers to generate the grasp-configuration features. Then we concatenate
the grasp-configuration features and the final object features
and pass them through two fully connected layers followed
by a sigmoid output layer to generate the grasp-success probability. We apply batch normalization for all convolutional
and fully connected layers except the output layer. We train
our voxel-based classifier using the cross-entropy loss.
To generate the voxel grid, we first segment the object
from the 3D pointcloud by fitting a plane to the table using
random sample consensus [31], [32] and extracting the
points above the table. We then estimate the first and second
principle axes of the segmented object to create a right-handed object-reference frame aligned relative to the world frame.
We compute the object size along the three coordinates of the
object-reference frame to construct the object-size vector. We
then generate a 32 # 32 # 32 voxel grid oriented about this
reference frame. We define the center of the voxel grid to be
the centroid of the points in the object segmentation. More
details of the pointcloud voxelization can be seen in the section "Grasp Model Training."
Voxel-Based Grasp Prior Networks
To model the grasp-configuration distribution based on the
geometry of the object of interest, we construct an MDN as our
object-conditional prior. Given its input, an MDN predicts the
parameters (means, covariance, and mixing weights) of a
GMM as output. Our MDN takes the object voxel grid and
object-size vector as inputs and predicts the parameters of a
GMM modeling a probability distribution over grasp configurations. Thus, the MDN learns to model the conditional probability distribution p ^i z, Uh, where U defined the learned
weights of the MDN. We train the MDN over all grasp
attempts from the training set, i.e., the learned distribution
models the probability that the specific grasp i being evaluated was observed at training, given the current object.
The MDN generates its object-feature representation using
the same subnetwork structure as the voxel-based classifier.
The MDN then passes the final object-feature representation
through two fully connected layers with rectified linear unit
activations. These two fully connected layers have 128 and 32
neurons respectively. Finally, the fully connected output layer
predicts the weights, mean, and diagonal covariance of the
mixture distribution over grasp configuration. Figure 1(b)
shows the architecture of our MDN. We apply batch normalization for all layers of the MDN except the output layer. We
train our voxel-based MDN using the negative log likelihood
loss. Figure 2 shows the mean grasp configuration of each

http://scikit-learn.org/stable/index.html http://scikit-learn.org/stable/index.html

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020