IEEE Robotics & Automation Magazine - December 2019 - 66

cognitive load from the operator. Some objects, such as tools,
require functional grasps: they must be grasped in a specific
way to enable their use. This requirement poses challenges to
the autonomy. Additional challenges arise for bimanual tasks,
which force the execution of collision-free synchronous
grasping motions.
The presented autonomous grasping pipeline enables
bimanual functional grasping of unknown objects if other
instances of this object
class are known (Figure 6). Input is RGB-D
dat a. First, s emantic
Two-dimensional height
s e g mentation-based
on Refine-Net with addimaps are inadequate for
tional postprocessing-
representing terrains, such finds object contours and
outputs corresponding
point clouds. It is trained
as expanses of fine gravel.
on synthetic scenes with
objects coming from turntable captures or CAD
mesh renderings. Subsequently, the object pose is estimated from the extracted point cloud, and a grasp is generated. Grasp generation is based on a nonrigid registration
that incorporates class-specific information embedded in
a low-dimensional shape space for the object category. To
learn the shape space of a category, object point clouds
are registered toward a single canonical instance of the
category [25], [26]. During inference, we optimize a
shape that best matches the observed point cloud. Finally,
individual grasping motions for each arm, which are

Semantic Segmentation

known for the canonical model, are warped to fit the
inferred object shape. Arm trajectories are subsequently
optimized by incorporating collision avoidance, joint limits, end-effector orientation, joint torques, and trajectory
duration [27]. Since the optimization is performed in joint
space, multiple end effectors can be easily added to the
optimization problem by incorporating additional terms
in the cost function. For manipulating an object (once it
is grasped) with multiple end effectors simultaneously, an additional term that penalizes violations of the
closed kinematics chain constraint is added. The planned
motion is finally presented to the operator for verification. The execution of the complete planning pipeline
takes fewer than 7 s.
Evaluation
The final CENTAURO system was evaluated using a systematic benchmark test at the facilities of KHG. Tasks were
designed based on KHG's knowledge about real-world
disaster-response missions. While the intermediate evaluation reported in [10] was carried out at the same facilities,
one year of further intensive system development significantly extended and advanced the CENTAURO system.
Hence, the described tasks in this article are designed to be
considerably more challenging to demonstrate the significantly increased real-world applicability of the final CENTAURO system.
All tasks were performed without previous training and
with the operator station located in a separated room, thus
preventing direct visual contact. The robot operated mostly
untethered, relying on power supplied through its battery and

Pose Estimation

Grasp Generation

Driller
0.969098

Warped
Frames

Observed
Point Cloud
(a)
Execution

(b)
Operator Validation

(f)

(e)

(d)

Figure 6. Photos and illustrations showing the autonomous manipulation pipeline. (a) From RGB-D images perceived by the robot,
the semantic segmentation finds the contour of the object-a drill, in this case-and (b) extracts the point cloud. Based on the contour
and the point cloud, the 6D pose of the object is estimated. (c) Subsequently, the segmented point cloud is registered nonrigidly
using the shape space of the object category, and control poses (e.g., pregrasp and grasp poses) for both arms are warped from the
grasping motion of the canonical object. (d) The final joint trajectory is generated by a trajectory optimizer that guarantees collisionfree motions. (e) The resulting motion is verified by the operator and (f) finally executed.

IEEE ROBOTICS & AUTOMATION MAGAZINE

DECEMBER 2019

IEEE Robotics & Automation Magazine - December 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2019