Signal Processing - September 2016 - 19

cameras, this allows image deblurring, refocusing [20], and
rise of miniature computational photography since there now
depth sensing [18].
exists useful fabrication strategies [3]. However, most of the
The key lesson learned by these early computational phoprevious efforts in this area have been to create optics for
tography researchers was that important scientific questions
generating sharp, high-quality imagery. For example, a
involved the coded aperture patterns and
variety of techniques exist to create microthe related decoding algorithms for images
lenses by taking advantage of surface tenComputational
captured under these apertures. Making the
sion properties of PDMS and other
photography is about more coded aperture itself enjoyed the support
materials that are heated and form lens
than just capturing images of relatively established approaches, espeshapes when in liquid form. Microlenses
now form an integral part of many smartbut is also about exploiting cially if the coded aperture in question was
binary. At the millimeter scale, laser printphone cameras, as they collect light within
the image-formation
ing provided the required resolution. For
each pixel on the sensor. In research, a
process to extract even
smaller and more complex systems, phogoal has been to create miniature optics
more information from
tolithography techniques such as the 1 μm
that mimic insect eyes [16] or that offer
the world.
Heidelberg photomask writer could easily
shape control of microlenses [6].
do the job. Therefore, many computational
While these previous efforts focus on
photography researchers became the new customers of the
the extremely useful goal of creating high-quality images,
existing national nanotechnology infrastructure built during
they cannot provide the full story. Computational photograthe 1990s and 2000s.
phy is about more than just capturing images but is also about
The plenoptic designs created by the aforementioned phoexploiting the image formation process to extract even more
tolithography techniques were static and could not be changed
information from the world. It includes sampling the lightover time. To create programmable optics, researchers
field, encoding the incoming light-rays and even analysis of
took advantage of the wide availability of display related
the scene itself through filtering and optical convolutions. The
technologies for manipulating light, such as liquid crystal
fabrication technologies for creating micro-optics are useful
displays or digital micromirror devices that allow either confor making computational cameras at small scales, but the
trolled sampling of the light-field or processing of informadesign tools available require updating. For example, ray traction for computer vision and image processing. Initially, these
ing softwares that model aberrations and image blurring and
efforts required systems engineering; for example, in [19], the
that assume a plano-parallel scene model are still the norm.
researchers hacked a Texas Instruments DLP projector, using
However, geometric distortions reduce for small optics, and,
it as a camera instead of a projector and whose "projected"
instead, diffraction becomes important, posing both a chalpatterns became the camera's coded aperture. Today, almost
lenge and an opportunity, as we will see in the next section.
ten years later, the Texas Instruments developer kit is affordWide-angle fields of view (FOV) become important since
able enough that such hacking is no longer common. In fact,
narrow FOV miniature platforms must move to capture the
this availability has resulted in some of the most visible sucsurrounding visual field, which has power costs. However,
cesses of compressive sensing [27] and continues to impact
wide-angle optics, while well understood at large scales, are
vision and imaging. This is a past example of the evolution
not easily manufactured at the miniature scale. For example,
and commodification of key technologies that we believe will
miniature fish-eye lenses consist of multiple optical elehappen in the future for many of the related areas summaments at cm scales with only 120° FOV being demonstratrized in Figure 2.
ed. Curved mirrors allow panoramic imaging for computer
vision applications and have no dispersion related problems;
unfortunately, to the best of our knowledge, the state of the
A first wave of computational
art for miniature mirrors does not appear to have a greater
photography in the small
FOV than 45° [11].
There has been a recent surge of miniature computational
cameras, and some of these are illustrated in Figure 3. The
previous efforts we discuss here may lack integration, but
Plenoptic designs in computational photography
they represent a new line of thinking that seeks to merge the
Fourier optics [12] involves building optical systems to impleintertwined technologies of plenoptic designs, miniature
ment computations like Fourier transforms by, among other
optics, and computational sensing in hardware and algothings, designing point spread functions (PSFs). For decades,
rithms to create new types of cameras. Figure 3 depicts these
such optical processing research resulted in the use of both
on an axis of optical size and power consumption. Each of
coherent light and partially coherent light to build computing
the authors cited reported their sensors' optical size, but, calplatforms that were meant to compete with silicon-based
culating the power footprint was more challenging since it is
computers. Ten years ago, controllable PSFs began to appear
subject to interpretation and can change depending on the
in computer vision and computer graphics communities,
task at hand. For example, the raw images from a sensor
where attenuating templates, assorted pixels and plenoptic
could be used for optical flow directly, without much power
designs created by standard photolithographic techniques, filconsumption. However, the same sensor might require
tered scene radiance before measurement. For consumer
IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016