Signal Processing - May 2017 - 61

In [31], Taguchi et al. propose another scalable depth-sensing
An early work of space-time coding is described by Zhang
method, where they also use a De Bruijn color stripe as
et al. in [29]. They first design a color stripe pattern by realizing
the base pattern similar to [29] and obtain a temporally peria De Bruijn sequence as shown in Figure 6(a), which supports
odical pattern set by shifting the base pattern regularly over
single-frame depth reconstruction for high-speed scans of moving
time. The whole pattern set contains eight patterns and
objects. By smoothing the color-stripe pattern with a Gaussian filallows for decoding with a f lexible number of frames
ter and projecting its shifted copies (in the direction perpendicular
(e.g., one, two, four, or eight) at every pixel. In other words,
to the stripe) over time, multiframe depth reconstruction is then
motion-aware spatiotemporal window sesupported for high-accuracy scans of static
lection is supported during decoding. The
scenes. For single-frame reconstruction, they
Once the scene becomes
spatial window size decreases as more temdevelop a multipass dynamic programming
dynamic, human attention
poral frames are used. If eight frames are
algorithm that eliminates global smoothwill be largely attracted by
used, the spatial window size shrinks to only
ness assumptions and strict ordering conthe object motion and thus
one pixel. One highlight of this work is that
straints present in previous formulations. For
be less sensitive to the
pixel-wise optimal depth reconstruction is
multiframe reconstruction, space-time analyobject details.
supported based on the plane-sweeping alsis is conducted at each sensor pixel to obtain
gorithm. That is, for each pixel, the optiinterframe depth localization. Using a short
mal depth value should give the maximum matching score in
sequence of seven time-shifted stripe patterns, the accuracy of the
terms of the normalized cross-correlation. The matching score
multiframe reconstructed depth map is significantly improved for
is computed between the captured images and the projected
static scenes compared with the approach that first obtains seven
patterns, using all possible spatiotemporal window sizes at a
single-frame reconstructed depth maps independently and then
given depth layer. This space-time coding can handle dynamic
combines them into a high-resolution one, which demonstrates
scenes with different motions in different parts of the scene
the effectiveness of space-time analysis. Still, this work is one step
and improves the accuracy of depth measurement for static or
away from scalable depth sensing, as it does not support the transislowly moving parts.
tion between the two types of reconstruction.
As mentioned previously, phase-shifted fringes proThe first embodiment of scalable depth sensing is realvide higher accuracy depth measurement than other timeized by Ishii et al. in [30]. As shown in Figure 6(b), the base
coding methods, while the pseudorandom speckle pattern
pattern is periodical in one direction (vertical for example)
provides more robust depth measurement than other spacewith a period of N, while each column represents a 1-D Gray
coding methods. Combining these two distinct components,
code pattern. By shifting the base pattern N - 1 times in the
Zhang et al. introduce a set of hybrid patterns for scalable
vertical direction, one can get a set of patterns that are both
depth sensing in [32], where the nth(n = 1, 2, 3) pattern can
spatially and temporally periodical based on the Gray code,
which thus enables space-time coding with a flexible number
be described as
of frames. This work takes N = 8 as an example, so there
are eight decoding types: the first type uses eight codewords
P nH (x, y) = B H (x, y) Z (x, y) + m ;C + A cos c 2r f y + 2r n mE.
3
along the time dimension in eight different frames, the sec(10)
ond type adds one more codeword in space from the latest
frame and removes the dependency on the earliest frame,
Compared to (6), the main difference here is that the amplitude
and so on. Finally, the eighth type uses all eight codewords
of fringe needs to be reduced to support single-frame reconstrucin space from the latest frame and gets rid of the dependency
tion from the speckle, and m ! [0, 1] denotes the percentage of
on previous frames. Obviously, the first decoding type is
intensity dynamic range that the fringe component occupies. It
effective for measuring static objects, while the eighth type is
has been experimentally investigated that m ! [0.2, 0.4] can
effective for measuring moving objects. The adaptive selecprovide competitive accuracy of depth measurement in both
tion of decoding types n (1 # n # N)
is determined based on the frame
differencing features that detect the
decoding errors e caused by motion.
Specifically, n is proportional to e. In
other words, larger decoding types that
are robust to motion are selected when
e increases, and smaller decoding
types that enable accurate measurement are selected when e decreases.
In this way, scalable depth sensing is
(a)
(b)
(c)
elegantly realized with a smooth transition between single-frame and multi- FIGURE 6. Base patterns of space-time coding: (a) a De Bruijn color stripe, (b) a periodical Gray code,
frame depth reconstruction.
and (c) a hybrid speckle and fringe. (Images (a) and (b) are from [29] and [30], respectively.)
IEEE SIgnal ProcESSIng MagazInE

May 2017

Table of Contents for the Digital Edition of Signal Processing - May 2017