Signal Processing - September 2016 - 46

analog-to-digital image conversion. The classical Weiner filter
was used for image enhancement. Transform-domain coding utilizing discrete cosine and wavelet transforms played a pivotal
role in JPEG compression. Over the last few decades, with the
advent of the wavelet transform and compressed sensing theory, the field of signal processing has undergone a philosophical
reformation. A field that was once heavily reliant on smoothness
assumptions now uses principles inspired by the notion of sparsity. Until the last decade or so, the fields of image sensor technology and signal processing ran in parallel to one another with
minimal interaction or exchange of know-how. However, there
has recently been a growing trend toward the coherent codesign
of sensors and algorithms: this is the theme of the emerging area
of computational imaging or computational photography [1].
Practitioners of this computational imaging/photography
ideology have devised many solutions that were previously not
possible when individually adding functionality to the sensor
architecture or using a more sophisticated processing algorithm.
For example, consider the problem of motion deblurring that
arises in conventional imaging. Whenever an object moves during sensor exposure, it causes pixels to smear across the frame,
resulting in a blurred image. In the context of signal processing,
this is an ill-posed problem that has been well explored within
the theme of deconvolution. The key problem is that the exposure
time defines a temporal filter, which is essentially a box filter that
annihilates any high-pass, spatial information. Consequently,
algorithmic sophistication alone is not enough. Blurring may be
avoided by a shorter exposure time, but this comes at the expense
of low signal-to-noise ratio (SNR). It is clear that neither deconvolution nor the sensor level adaptation in itself suffice for a solution to the deblurring problem. The distinct role of computational
imaging emerges when one considers the so-called flutter-shutter approach [2]. This approach involves a codesign of sensor and
algorithms: in contrast to traditional imaging methods, which
require the shutter to be fully open during the exposure time, the
flutter-shutter method flutters the shutter on and off in a binary,
pseudo-random sequence. This sequence converts the noninvertible box-filter into an invertible one and, based on the choice of
pseudo-random sequence, the corresponding deconvolution filter may be devised. Beyond deblurring in consumer imaging, the
flutter-shutter approach is also useful in bioimaging [3], where
the imaging sensor may not be fast enough to capture flowing
structures, such as blood cells. Other notable examples of the
computational imaging philosophy are high-dynamic-range
imaging [4], light-field imaging [5], [6], single-pixel imaging [7],
and Fourier ptychographic microscopy [8].
For the most part, image sensor design, signal/image processing, and computational imaging have largely been restricted to
two-dimensional (2-D) scenes. However, a true and richer representation of the environment around us lives in a 3-D space. Capturing 3-D information of a scene offers unparalleled benefits in
accuracy and capabilities and is surely the future in many areas.
This necessitates development of imaging modalities capable of
recording 3-D images.
A number of methods have been developed for the purpose of
3-D imaging. An overview of the main techniques is presented in
46

"An Overview of 3-D Imaging Techniques." Of all the 3-D capture techniques, the ToF method has arguably attracted the most
commercial and scientific interest in the last couple of years;
there has been a surge of research toward improving both the
sensor design as well as the algorithms used for processing 3-D
images. ToF imaging is the theme of this article, and we take a
deep dive into the topic in the following section.

The time-of-flight revolution
The ToF principle exploits the idea that distance and time are
proportional quantities. As the name suggests, ToF is the roundtrip time between the source and the destination taken by a particle or a wave. Hence, knowing one entity is equivalent to
knowing the other. Nature is replete with examples that rely on
the ToF principle. Bats, dolphins, and visually impaired human
beings use the ToF principle for navigational purposes.
Chronologically, the use of sound waves superseded the use
of electromagnetic waves. Humans have used stones to estimate the depth of wells for millennia. The earliest work on
using light waves for measuring ToF dates back to an experiment conducted by Galileo, who was interested in estimating
the speed of light. Unfortunately, his choice of distance (the
separation between two hills) did not lead to a conclusive
result. The Danish astronomer Ole Rømer overcame this difficulty by using planetary distances. About 200 years later, the
French physicist Hippolyte Fizeau was the first to precisely
estimate the speed of light. Through the discovery of the law
of the photoelectric effect by Albert Einstein in the 1900s and
the development of the electronic imaging sensors [charge
couple device (CCD)/complementary metal-oxide-semiconductor (CMOS)], we are now at a point where the accumulated
research efforts in the area of photonics and electronics have
culminated in mass-producible optical ToF sensors.
Contrary to conventional imaging sensors such as digital
cameras that produce 2-D images I ^ x, yh , ToF sensors capture
3-D images, I ^ x, y, zh . The unique ToF sensor produces two
images per exposure: an amplitude image and a depth image.
The amplitude image is the standard 2-D photograph, I ^ x, yh .
Each pixel on the depth image represents the corresponding
distance in the scene. The combination of the amplitude and
the depth image produces the 3-D image. Using our customdesigned ToF sensor, we show the amplitude, depth, and resulting 3-D images in Figure 1.
ToF-based 3-D imaging allows for applications that were
previously unexplored. One of the first results demonstrated
non-line-of-sight imaging capability [10]. This result-in parallel to "Doc" Edgerton's iconic Bullet Through Apple image
(see Figure 2)-led to ultrafast imaging of light packets at
an exorbitant frame rate of one trillion frames per second. A
flurry of follow-up work lead to results that allowed imaging
through scattered media [11], light-in-flight imaging [12], [13],
and 3-D imaging in extremely low light [14].
With the advent of 3-D sensing technology (most notably,
the Microsoft XBox One's Kinect) we can now replace a roomsized apparatus [10], moving sensors, and raster scan systems
[14] by miniaturized, cost-effective, real-time, and full-frame

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016