IEEE Robotics & Automation Magazine - December 2013 - 58

Figure 12. The LEGO Oasis demo using RGB-D recognition [21].
Near-real-time object recognition allows interesting human-computer
interactions using an overhead projector: a LEGO fire truck putting out
a virtual fire on a house.

objects and their orientations on the table, it can perform
interesting interactions using an overhead projector, such
as projecting fire to simulate a dragon breathing fire onto a
house. The success of this demo, shown at various places,
including the Consumer Electronics Show, illustrates the
robustness of RGB-D recognition and what possibilities
real-world object recognition could open up.
To Get Started Using RGB-D
One major advantage of using RGB-D cameras is that the
hardware is available to everyone, with a consumer price tag
that makes it practically appealing. While the focus of this
article is algorithm research, ease-of-use software development has also been at the center of attention for RGB-D perception. The PCL [22] is a central place where open-source
RGB-D software packages are being developed. They are easily accessible and closely integrated with the ROS platform
and the OpenNI framework. The basics of PCL can be found
in a recent tutorial published in IEEE Robotics and
Automation Magazine [23].
For RGB-D mapping and 3-D modeling, the RGB-D
mapping software of Henry et al. [5] has been made available as an ROS stack at the authors' Web site at the
University of Washington, enabling 3-D mapping of large
environments at the floor scale with a freely moving RGB-D
camera. The RGBD SLAM package from Freiburg has similar functions and is a part of ROS. The KinectFusion system
[10] has a real-time open-source implementation, available
in the trunk version of PCL, which allows highly detailed
modeling at the room scale.
For RGB-D object recognition, the RGB-D kernel
descriptor software [17] is available in both MATLAB and
C++ at the authors' Web site, producing state-of-the-art
accuracies on the RGB-D object data set [14], a publicly
available benchmark with densely sampled RGB-D views of
300 everyday objects. The real-time LINEMOD object detector is included in Open Source Computer Vision Library
(OpenCV), making use of both color and depth data. PCL
58

IEEE ROBOTICS & AUTOMATION MAGAZINE

DECEMBER 2013

and OpenCV also include a variety of other recognition
algorithms such as Speeded-Up Robust Features (SURF) for
color and Viewpoint Feature Histogram for depth.
There are a number of other growing software platforms
for RGB-D perception in addition to PCL and OpenNI.
The Microsoft Kinect software development kit (SDK) [24]
brings Kinect cameras and their capabilities to Windows
developers, such as the Xbox skeleton tracking [25] and
speech recognition. The Intel Perceptual Computing SDK
[26] is another hardware-software initiative that features
RGB-D-based gesture recognition and hand tracking
among other things, with a million-dollar call for creative
usage. Both platforms, backed by large corporations, provide an extensive set of basic functionalities and have been
attracting more and more software developers, redefining
the future of perceptual computing.
Conclusions
In this article we have discussed our recent work on RGB-D
perception: jointly using color and depth data in affordable
depth cameras for large-scale 3-D mapping and recognition.
Our efforts are part of a much bigger picture of utilizing
RGB-D devices for visual perception across multiple research
domains. Much progress has been made since the release of
Kinect, showing the great potential of RGB-D perception for
a wide range of problems, from 3-D mapping and recognition
to manipulation and human-robot interaction. Combining
color and dense depth at real time, with a consumer price tag,
RGB-D cameras provide numerous advantages over optical
cameras and laser rangefinders.
We have found that RGB-D perception is much more
robust and often more efficient than using RGB alone. An
RGB-D camera is fundamentally better than a traditional
optical camera with the same resolution in that: 1) it largely
recovers the 3-D structure of the world without losing the
structure in the projection to 2-D images, and 2) its depth
channel is largely independent of ambient lighting. Although
ultimately every perception problem is solvable using an
optical camera (or a stereo pair), in practice, RGB-D cameras make it much easier to develop robust real-time solutions that roboticists can use as building blocks to explore
their own research problems.
We have also found that RGB-D perception is more discriminating, providing richer information than obtained using
depth alone. The Kinect pose tracker and the KinectFusion
3-D modeler do not use color; such depth-only approaches
could be attractive if depth alone contained sufficient cues. On
the other hand, the color channel typically has higher resolution and higher granularity, is not limited by range, and is
much easier to improve in hardware than the depth channel.
Color and depth channels are time synchronized in the
PrimeSense/Asus cameras and possibly in future versions of
Kinect. We expect more and more color + depth approaches
as we target harder problems that require richer inputs.
The field of RGB-D perception is quickly evolving, and so
are the cameras themselves. Depth resolution, accuracy,

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2013