Signal Processing - May 2017 - 56

sible with traditional depth cameras, or reduce the cost or size of
devices, will largely rely on high-performance commodity
camera elements.
depth cameras. The recently released Microsoft HoloLens [8]
For the completeness of this review, we first briefly inand Google Tango [9] are two good examples.
troduce traditional structured light techniques, through
Inspired by the success and philosophy of the Kinect, much
which it is easier to understand the merits of computational
research has been done in the past few years targeting realtechniques. Traditional structured light techniques can be
time, high-performance depth sensing. These works share a
roughly divided into two categories according to the pattern
common insight that the codesign of image sensor systems and
design strategy: time coding and space coding. Time-coding
signal processing algorithms is essential to achieve a superior
methods project multiple patterns sequentially to identify
performance. Such codesign is called computational depth
each point in the scene with a time series, e.g., binary code
sensing. In this article, we introduce this important concept and
[12], Gray code [13], N-ary code [14],
provide an overview of the latest represenand phase-shifted fringe [15]. Due to the
tative techniques. By bringing together and
Due to its promising and
pixel-independent encoding, time-coding
analyzing interdisciplinary research from
reliable performance, the
methods can provide high-accuracy depth
signal processing, computer vision, and
structured light approach
with simple decoding algorithms. Howevoptics communities, our goal is to shed light
has been widely adopted
er, time coding cannot deal with dynamic
on the development of future commodity
for three-dimensional
scenes unless expensive high-speed harddepth cameras, which is a potential great
scanning purposes.
ware is used [16]. In contrast, space-coding
interest to a broad audience. Specifically,
methods using a single pattern are suitable
this article will focus mainly on the strucfor 3-D capture of dynamic scenes. Generally, this pattern
tured light approach, which provides a large degree of freedom
needs to be designed in a way that each point is uniquely enfor the design of depth cameras. A recent review of ToF camcoded by its neighborhood, e.g., De Bruijn sequences [17],
eras is given in [10], and another on light-field cameras is given
M-arrays [18], and color-coded [19] and symbol-coded [20]
in [11]. Also, a comprehensive review on traditional structured
patterns. Nevertheless, space-coding methods rely on an aslight cameras can be found in [4]. This article will be complesumption of local depth smoothness that will be violated by
mentary to these reviews.
abrupt depth changes. Thus, the accuracy of depth obtained
from space coding is limited.
Computational depth sensing
We still take the Kinect as an example to further explain
For ordinary digital cameras, what you see is what you get
the concept of computational depth sensing. The pattern used
under appropriate ambient illumination. By contrast, depth camin the Kinect belongs to space coding for structured light. In
eras do not sense depth information directly, but, rather, through
traditional space-coding methods, color or grayscale informaeither the space deformation or the time delay of light signals.
tion is generally used. These color or grayscale patterns need
In this sense, computation is actually indispensable for all depth
to be emitted by projectors. While projectors are often expencameras. In this article, however, computational depth sensing is
sive, bulky, low-energy, and offer a small depth of field, the
defined as the redesign of image sensor systems or elements
emitted light is also susceptible to ambient illumination and
followed by advanced signal processing algorithms.
scene albedo. All of these issues hinder the practical applicaComputational depth sensing can improve the capabilities of
tion of space-coding depth cameras. The revolutionary change
traditional depth cameras, introduce features that were not posin the Kinect is to use a tiny laser-diffuser emitter to replace
the projector. The diffuser bears a binary, pseudorandom
speckle pattern, and the laser emits infrared light. Therefore,
while being invisible to human eyes, an infrared speckle pattern is generated and projected onto the scene, as shown in
Figure 1. Due to the well-designed pseudorandom distribution
of the bright dots, this speckle pattern has high distinguishability when observed in a local window. Compared with the
projector, the laser-diffuser emitter has obvious advantages:
low cost, compact size, high energy, and large depth of field.
Moreover, compared with the projector-generated color or
grayscale patterns, this binary, infrared speckle pattern is
robust to ambient illumination and scene albedo. All of these
issues contribute to the reliable performance of the Kinect
depth camera.
Besides the unique optics design, the Kinect also adopts
an efficient signal processing algorithm, that is, how to compute depth once the deformed speckle pattern is captured.
FIGURE 1. An infrared camera image of part of the speckle pattern
The triangulation principle is used here. A reference image
projected by the Kinect.
56

IEEE SIgnal ProcESSIng MagazInE

May 2017

Table of Contents for the Digital Edition of Signal Processing - May 2017