Signal Processing - September 2016 - 17

From [29]

From [5]

From [8]

From [1]

Light Sources

Active
Optical
Elements

Optical Filters

(a)

Sensor Array

Optical Filters

Passive
Optical
Elements

Photodetector
Patterns

Hardware
and
Output
Software

(b)

Figure 1. Miniature sensors: a new frontier for computational photography. In (a), a few motivating examples (images used courtesy of [1], [5], [8], and
[29]) illustrate the coming, new wave of small machines that are transforming surveillance, medicine, sensor networks, agriculture, and other fields.
Some, such as [1], are commercially available. However, due to restrictive power/mass budgets, none of these systems have cameras, let alone computational photography capability. If these devices could visually sense their environment, their impact would greatly increase. In this survey article, we cover
relevant work in computational photography, compressive sensing, micro/nano optics, sensor fabrication, miniature displays, and embedded computer
vision that together are defining the subdiscipline of computational photography in the small. In (b) we show the overall framework of such a miniature
computational camera, where every sensor aspect, from optics to computing, is influenced by the visual task at hand.

platform. The design of each sensor can be optimized so that
the computation is distributed across all aspects of the device,
including passive optics to modulate the incoming light, active
optics to project patterns onto the scene, optical filters for either
polarization or wavelength as well as accompanying embedded hardware and optimized software. This comprehensive
strategy can address the problem of achieving computational
photography on compact devices.

Converging miniature sensor technologies:
A brief history
In the last two decades, a few billion cameras became available to a large portion of humanity. This created a surge of
interest and accompanying progress in a variety of imaging
related technologies including, to name just a few, efficient
hardware, small optical designs, miniature light-field sensors,
and compact active illumination and displays.
We focus here on a brief history of three technologies in
particular that have built the foundation for computational
photography in the small. The first is the maturing of embedded vision sensing technologies, which includes both massproduced low-power computing platforms from the mobile

revolution as well as specialized systems that intentionally blur
the lines between computing hardware and sensing. The second is the impact of miniature optics for visual sensing, where
display and imaging optics that were previously only created in
research labs are now widely available. The third is the recent
application of plenoptic designs to consumer cameras to allow
for increased postprocessing control of photography.
Taken together, these fields have created the opportunity to
make a new type of camera, as illustrated in Figure 2. This is
a camera in which the visual task at hand can influence every
aspect of the sensor, from the scene illumination and imaging
optics to the sensing electronics and on-board processing. This
allows for truly task-specific sensors that can extract every
possible size, power, and mass efficiency from the system and
can enable miniature computational cameras.

Embedded vision sensing and the mobile revolution
Processing images and video in real time on hand-held devices
over the last two decades has resulted in a mature infrastructure
for low-power vision and imaging. Dedicated imaging application-specific integrated circuits (ASICs), consisting of digital signal processors (DSPs), field-programmable gate arrays (FPGAs)

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016