IEEE Robotics & Automation Magazine - December 2013 - 54

natural and easy solution to the problem. Our studies show
that not only can we reliably scan large environments with
RGB-D, we can do so efficiently in near real time. This is confirmed in the work of Engelhard et al. [8], whose open-source
RGBD SLAM (simultaneous localization and mapping) package in the robot operating system (ROS) has been used in a
number of robotics projects.
One problem closely related to environment mapping is
the modeling of 3-D objects. Krainin et al. [9] showed an
example in the context of robot manipulation, where a
robot rotates and studies a novel object in its hand, incrementally building and updating a 3-D model of the object.
Robot hand motion is guided by next-best-view selection
based on information gain computed from a volumetric
model of the object. If needed, the robot places the object
back on the table and regrasps it in order to enable new
viewpoints. With knowledge of its own arm/hand, and
computing articulated ICP tracking of the arm and the
object jointly, the robot can acquire a good 3-D model of
the object using an RGB-D camera. Such autonomous
object modeling is a first step toward enabling robots to
adapt to unconstrained environments.
It is interesting to compare the RGB-D mapping system of
Henry et al. [5] to KinectFusion [10], a more recent system
developed at Microsoft that shows fine details for modeling
room-size spaces. KinectFusion uses a depth-only solution to
visual odometry, partly because the color and depth frames in
Kinect are not time synchronized. Running a highly optimized
version of ICP on graphics processing units, KinectFusion,
along with its open-source implementation [in the point cloud
library (PCL)], can run in real time near 30 frames/s. The
robustness of ICP is greatly improved by aligning an incoming
RGB-D frame to the partial 3-D model in a volumetric representation (instead of to the previous frame). In comparison, the
RGB-D mapping system targets mapping at a much larger
scale, as it has stronger loop closure capabilities and a less
detailed 3-D representation.
The ability to build and update 3-D maps of environments can have major impacts on robotics research and
applications, not only in navigation and telepresence but
also in semantic mapping and manipulation. A detailed
3-D map serves as a solid basis for a robot to understand its
surroundings with higher level semantic concepts. As an
example, Herbst et al. used RGB-D mapping and scene differencing for object discovery, using scene changes over
time to detect movable objects [11]. Discovering and modeling unknown objects is a crucial skill if a robot is to be
deployed in any real-world environment. Threedimensional mapping has also been used in 3-D scene
understanding and labeling [12], [13], which is further discussed in the next section.
RGB-D: 3-D Object and Scene Recognition
Modeling an environment in terms of 3-D geometry and
color is only the first step toward understanding it. For a robot
to interact with the environment, object recognition is the
54

IEEE ROBOTICS & AUTOMATION MAGAZINE

DECEMBER 2013

key; the robot needs to know what and where the objects are
in a complex environment and what to do with them.
Our studies show that RGB-D perception has many advantages for object recognition. Comparing with image-only recognition, RGB-D provides 3-D shape data and makes it feasible to detect objects in a cluttered background [16].
Comparing with point cloud recognition, RGB-D uses color in
addition to shape, leading to richer and more distinctive features especially for object instance recognition [17].
Combining robustness and discriminative power with efficiency, RGB-D object recognition achieves high accuracy classifying hundreds of objects [15] and is quickly becoming practical for analyzing complex scenes in real-world settings [18].
Recognition of Everyday Objects
For a robot to operate in the same environment where people
live, its recognition task is mainly to find and classify objects
that people use in their daily activities. The recognition task
also spans multiple levels of specificity, such as category recognition (Is this a coffee mug?), instance recognition (Is this
Kevin's coffee mug?), and pose recognition (Is the mug with
the handle facing left?). A robot would need to be able to
answer all these questions.
There were few object data sets in robotics or computer
vision that covered a large number of household objects, and
none existed for RGB-D data. The first task was to create such
a data set on which features and algorithms could be evaluated. Lai et al. [14] collected an RGB-D object data set that
captured 51 object categories and 300 object instances using a
turntable, with a total of 250,000 RGB-D frames. Figure 6(a)
shows some of the objects included. This data set allowed
them to carry out empirical studies of state-of-the-art features, such as SIFT, histogram of oriented gradients (HOG),
and Spin Images and classifiers including linear support vector machine (SVM), kernel SVM, and random forest [14].
What image and depth features should we use for RGB-D
recognition? Comparing with well-studied image features,
depth features for recognition were underdeveloped; standard
features such as Spin Images were not designed for viewbased recognition. The work of Bo et al. [17] extended kernel
descriptors, previously developed for image classification, to
the depth domain. Kernel descriptors are a flexible framework that constructs a local descriptor from any pixelwise
similarity function using kernel approximation. Five different
depth kernel descriptors were developed, based on gradient,
local binary pattern, surface normal, size, and kernel signature
(using eigenvalues in kernel PCA). These kernel descriptors
were shown to perform much better than standard features.
More recently, Bo et al. [15] pursued a promising line of
work that learned feature representations from scratch using
sparse coding. In place of hand-designed features such as
kernel descriptors, they presented a feature-learning architecture that automatically and efficiently learned local
descriptors from data using K-SVD and orthogonal matching pursuit. Given a set of image patches Y = 6 y 1, f, y n@,
K-SVD jointly finds a dictionary D = 6d 1, f, d m@ and an

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2013