Signal Processing - May 2017 - 66

and the hardware equipped in depth cameras should support
of space-time multiplexing promotes the usage of commodhighly parallel computing. While the processing for tradiity hardware while maintaining the measurement accuracy.
tional techniques such as space coding and time coding are
Specifically, it is worth mentioning that binary phase shifting is
parallelizable in principle, more sophisticated algorithms
a promising approach to enable the design of high-performance
may be required for an improved performance in computacommodity depth cameras, as binary patterns can be easily
tional depth sensing, which are not necesgenerated using relatively inexpensive DLP
sarily easy to be parallelized. On the other
projectors or laser-diffuser emitters that cost
In this mobile-first world,
hand, previous works generally demonmuch less than DLP projectors.
high-accuracy depth
strate parallel implementation on CPU/
Another common insight behind these
information would enable a GPU, yet seldom on resource-constrained
techniques is that interdisciplinary design of
great deal of applications if hardware such as field-programmable gate
system principles and components could break
available on mobile phones. arrays [45]. The latter, however, is closer
the performance limit of traditional depth
to the real case for the deployment of
sensing. From this perspective, phase period
commodity depth cameras. Therefore, the design of parallel
coding implicitly uses space coding to solve the phase ambiguity
signal processing algorithms adapted to resource-constrained
in phase shifting (a kind of time coding) without using a large numhardware should receive more attention.
ber of patterns, space-time coding explicitly integrates advantages
of space coding and time coding within a scalable depth-sensing
paradigm, and sensor fusion directly combines different types of
Depth cameras on mobile phones
depth sensors for an improved performance while maintaining
In this mobile-first world, high-accuracy depth information
backward compatibility. Furthermore, binary phase shifting borwould enable a great deal of applications if available on
rows the idea of dithering from digital image processing to achieve
mobile phones. However, it is difficult to integrate active
high-accuracy depth measurement with elegantly simulated sinudepth cameras into mobile phones, mainly due to the physical
soidal fringes from binary patterns.
size of the light source. Although the laser-diffuser emitter
Essentially, the aforementioned insights can be summarized
used in the Kinect could meet the rigorous requirements,
into two aspects: advanced signal processing algorithms and
safety is another issue to be considered, as lasers may not be
redesign of image sensor systems or elements, which together
preferred for near body usage. Fortunately, the advancement
mark the era of computational depth sensing. Along with the
of projector technology could relieve this problem. The miniaever-increasing computing power and the advancement of light
turized version of DLP projectors would soon be available on
source technology, we believe computational techniques will
mobile phones [46]. The Lenovo Phab 2 Pro [47], the first
play a more and more important role in the development of
commodity Tango-powered mobile phone with a built-in ToF
future depth cameras.
depth camera, is already on the market. Still, practical issues
such as energy efficiency and measurement range need to be
taken into consideration for the development of depth cameras
Advanced issues
on mobile phones.

Open problems

Despite the rapid progress made in the field of computational
depth sensing, there are still some major challenges that are
considered important open problems.

Outdoor and global illumination
One issue that restricts the application of active depth cameras
is undesired illumination. A typical scenario is outdoors, where
the presence of strong sunlight severely interferes with active
illumination. Attempts to overcome this difficulty usually have
side effects of degraded measurement accuracy or speed [43].
Another kind of undesired illumination, called global illumination, refers to interreflections, surface scattering, and other
effects that are not directly from the light source. Global illumination effects frequently pose obstacles for active depth cameras, and avoiding them has been investigated in the literature
[44]. Nevertheless, it remains challenging to maintain measurement accuracy, speed, and reliability at the same time.

Hardware-friendly algorithm design
For real-time, high-resolution depth sensing, parallel computing is indispensable. Both the signal processing algorithm
66

Promising applications
With improved performance and reduced cost, commodity
depth cameras have found their applications in a number of
emerging scenarios. Here we discuss several application
scenarios that are expected to have a large impact in the future.

VR/AR
VR and AR are becoming extremely popular. However, the
richness of the content is one bottleneck for this industry. As
3-D information is necessary for generating an immersive
user experience, depth cameras play an indispensable role in
this scenario. Equipped with a high-performance depth camera, VR/AR devices can easily obtain the 3-D information of
the environment in real time and then render the reality as
one wishes. That is the working principle behind the
Microsoft HoloLens [8] and Google Tango [9]. To be competitive in the market, depth cameras on VR/AR devices
should have low cost, compact size, and large measurement
range. As can be expected, the growth of the VR/AR industry will greatly promote the development of commodity
depth cameras.

IEEE SIgnal ProcESSIng MagazInE

May 2017

Table of Contents for the Digital Edition of Signal Processing - May 2017