Signal Processing - May 2017 - 65

interfere with each other unless precise synchronization is
conducted. To this end, an interference-free synchronization
strategy is designed with minimal restrictions on the system,
after analyzing the properties of light signals emitted by the
ToF and phase-shifting sensors. Second, calibration is essential
for establishing an accurate mapping between the depth measurements from ToF and phase shifting. Thanks to the highresolution RGB camera that is well calibrated with the ToF
sensor in the Kinect Gen2, a cross-modal calibration strategy
is designed to reliably relate the depth measurements from ToF
and phase shifting. Based on the calibration and synchronization strategies, a 30-fps real-time depth-sensing system has
been developed with the assistance of a GPU. Figure 12 shows
some 3-D reconstruction results of fusion in comparison with
ToF and phase shifting alone. As can be seen, this fusion method produces accurate and robust depth with high resolution and
low delay, which successfully integrates the advantages of ToF
and phase shifting. Note these two scenes are both with textureless surfaces, which poses difficulty for the previous fusion
methods using passive stereo.

Summary
Now that we have reviewed the representative computational
depth-sensing techniques, let us summarize by taking a deeper
look into the common essence of these techniques and how
these techniques will facilitate the future development of highperformance commodity depth cameras.
A common insight behind most of the techniques discussed
previously is to improve the efficiency of space-time multiplexing in terms of the light signal usage, which is different from
traditional structured light cameras that explicitly separate
space coding and time coding for specific application scenarios
(dynamic capture or static scanning). Actually, to achieve the
goal of fast and accurate depth sensing, space-time multiplex-

(a)

Monocamera

Kinect Gen2

DLP Projector

FIGURE 11. The system of ToF and phase-shifting fusion.

ing is indispensable. From this perspective, phase period coding exploits the space redundancy in time-multiplexing signals
(i.e., phase-shifted fringes) to reduce the time dependency of
high-accuracy phase shifting, space-time coding exploits the
time redundancy in space-multiplexing signals (e.g., De Bruijn
sequence, Gray code, and speckle pattern) for improved accuracy of motion-insensitive regions in the scene, and sensor
fusion exploits the cross-modal redundancy to pursue even
higher efficiency of space-time multiplexing through precise
calibration and synchronization. On the other hand, cost is an
important consideration. For structured light cameras, DLP
projectors, especially high-speed ones, are usually the most
expensive components. Therefore, improving the efficiency

(b)

(c)

(d)

FIGURE 12. Three-dimensional reconstruction results of two scenes with textureless surfaces: (a) the color image, (b) the ToF results, (c) the results of
three-frequency phase shifting (nine patterns), and (d) the results of ToF and phase-shifting fusion (three patterns).
IEEE SIgnal ProcESSIng MagazInE

May 2017

Table of Contents for the Digital Edition of Signal Processing - May 2017