Signal Processing - May 2017 - 60

FIGURE 4. Online 3-D reconstruction results of a dynamic scene (facial expressions with black eyeglasses) using three speckle-embedded fringe patterns.

Table 1. A comparison of different phase period coding methods.
Period Codes

Image Number

3-D Reconstruction Speed

Computing Unit

Remarks

None [23]

40 fps at 532 × 500 pixels

2.8 GHz CPU

Single smooth surface

Unit-frequency fringe [25]

228 fps at 640 × 480 pixels

3.0 GHz CPU

Fringe amplitude reduced

De Bruijn sequence [26]

11 fps at 640 × 480 pixels

2.0 GHz CPU

Fringe amplitude reduced

Customized [27]

120 fps at 640 × 480 pixels

3.0 GHz CPU

Fringe frequency limited

Speckle [28]

15 fps at 640 × 480 pixels

2.83 GHz CPU

Phase segmentation required

the same, any newly captured image can be combined with
its preceding two images to produce a depth map. Therefore,
one can have a pseudo frame rate of 30 fps even when the
projector and the camera work at the same speed, by using a
sliding window strategy in the time dimension. However,
motion occurring during the acquisition time of three images
(100 ms in this example) is likely to cause measurement errors.
To address the motion problem, there are two main directions.
One is pattern design to further reduce the time dependency,
and the other is pattern generation to further reduce the hardware cost. In both cases, the accuracy of depth measurement
should be retained as much as possible. The efforts along these
two directions are detailed next.

Scene Capture

Depth Output
High Accuracy

It-N+1
.
.
.

Multiframe Reconstruction
No

DH,t

It-1
It

Motion?

Yes

Low Delay

Single-Frame
Reconstruction

DL,t

FIGURE 5. A general framework of scalable depth sensing with space-time
coding.

Space-time coding
Human eyes will focus on the object details in a static scene.
Once the scene becomes dynamic, human attention will be
largely attracted by the object motion and thus be less sensitive
to the object details. Therefore, it is possible to take advantage
of this property to integrate space coding and time coding
within a scalable depth-sensing paradigm. The basic idea is to
design a uniform space-time coding through which a highaccuracy depth map can be reconstructed from multiple consecutive frames for a temporally static scene, while a low-delay
depth map can be reconstructed from a single frame once the
scene becomes dynamic. The transition between these two
modes should be immediate and seamless.
Figure 5 shows a general framework for scalable depth sensing with space-time coding. Suppose there is a synchronized
projector-camera system working with structured light illumination, and the projected patterns have a temporal period of N.
The currently captured frame I t is first compared with the prior
N - 1 frames for motion detection. If the scene remains static
within N frames, the image set (I t -N +1, f, I t -1, I t) is processed
to produce a high-accuracy depth map D H, t . Otherwise, if the
scene becomes dynamic during the N frames, only I t is processed to produce a low-delay depth map D L, t . The scalability
can be further extended. On one hand, if the scene remains static
within n (1 # n # N) frames, the image set (I t -n +1, f, I t -1, I t)
can be processed to produce a depth map D n, t with progressively improved accuracy as the value of n increases. On the
other hand, a regionwise optimal depth map can be produced
through region-wise motion detection.

IEEE SIgnal ProcESSIng MagazInE

May 2017

Table of Contents for the Digital Edition of Signal Processing - May 2017