Signal Processing - September 2016 - 28

characterize when the spot can be well-approximated using
geometric (ray) optics.

Image formation
We characterize image formation using the geometric optics
model. While this approach largely ignores diffraction, the
resulting model is useful for the design and analysis of wellconditioned imaging architectures. Furthermore, the calibration procedure that we detail in subsequent sections can
account for unmodeled diffraction effects. For the simplicity
of notation, we assume a simplified 2-D world imaged by a
one-dimensional (1-D) mask and sensor. The extension to a
3-D world imaged by a 2-D mask and sensor is straightforward except where stated otherwise.
For a suitably defined scene irradiance vector x ! R N , the
scene-to-sensor mapping can be described using the linear set
of equations
y = Ux + e,

(1)

where U ! R M # N is the measurement matrix, y ! R M is the
image formed on the sensor, and e is measurement noise. This
model can be interpreted in two different ways:
1) Each sensor measures a weighted, linear combination of
light from multiple scene locations, and each row in U
encodes the weights for the respective sensor. For a scene
at infinity, the weights for two different sensor pixels simply differ by a translation of the mask pattern. As a consequence, the matrix U has a Toeplitz structure.
2) Every light source in the scene casts a shadow of the mask
on the sensor. Thus, the image formed on the sensor is a
superposition of shifted and scaled versions of the mask.
The shift and the scaling of the mask pattern encodes the
angle and distance of the light source onto the sensor.
These properties are invaluable in the design of masks
that provide near-optimal recovery under noise. Given the
image formation model in 1), our tasks are to formulate
an inversion algorithm that recovers the scene x from the
sensed image y and design mask patterns that achieve optimal recovery performance. We study both problems in the
subsequent sections.

Image reconstruction

Given the sensor measurements y ! R M and the measurement
matrix U, recovering x ! R N depends mainly on the rank of
the matrix U and its condition number. When rank (U) = N
and the matrix is well-conditioned, we can obtain an estimate
of x by solving the least-squares problem
min
Ux - y 22 ,
x

(2)

which has the closed-form solution xt LS = U + y = x + U + e ,
where U + = (U T U) -1 U T is the pseudoinverse of U. When
U is not well-conditioned, the least squares estimate xt LS suffers from noise amplification. When U is rank-deficient, the
matrix becomes singular and an estimate cannot be achieved.
28

In the ill-conditioned and rank-deficient cases, we can use
an image prior to regularize the inverse problem. Specifically,
instead of solving (2), we solve
min
y - Ux
x

2
2

+ m R (x),

(3)

where y - Ux 2 quantifies the data fidelity, R (x) is a regularization term that enforces an image prior, and m 2 0 controls
the tradeoff between fidelity and regularization. A popular
choice for the regularizer that is useful for noise-suppression
is Tikhonov regularization (also known as ridge regression)
via R (x) = x 22 .
Natural signals, such as images and videos, exhibit a host
of geometric properties including sparse gradients and sparse
coefficients in certain transform domains (e.g., Fourier or
wavelets). By enforcing these geometric properties, we can
suppress noise amplification as well as obtain unique solutions
even when U is rank-deficient (i.e., M < N). A pertinent example for image reconstruction is the total-variation (TV) model,
where the regularizer R (x) = x TV corresponds to the TV
of the image, which is computed from its gradients. Writing
the scene x as the 2-D image x (u, v) and defining g u = D u x
and g v = D v x as the u- and v-components, respectively, of the
spatial gradient of the image, the TV of the image is given by
R (x) = x

= / g u (u, v) 2 + g v (u, v) 2 .
u, v

The minimization (3) with a TV prior is convex and produces images with sparse gradients. A host of efficient techniques have been developed to obtain the solution. A range of
even more realistic image models have been developed (e.g.,
[30]), but the resulting optimization might not be convex.

FlatCam: A lensless imaging case study
To illustrate the design tradeoffs involved in a practical lensless camera design, we review the FlatCam [31], which was
inspired by the coded aperture imaging principles pioneered
in astronomical X-ray and gamma-ray imaging [2]-[4], [32]
(see "Coded Aperture in X-ray and Gamma-Ray Imaging").

Architectural overview
The FlatCam design achieves a large photosensitive area with
a thin form factor by replacing the lens with a coded, binary
mask. The thickness of the camera is minimized by placing
the mask almost immediately on top of a bare conventional
sensor array. The image formed on the sensor can be viewed
as a superposition of many pinhole images. An illustration of
the FlatCam design is presented in Figure 3. Light from all
points in the scene passes through a coded mask and forms a
multiplexed image on the sensor. A computational algorithm is
used to recover the original light distribution of the scene from
the sensor measurements.
The FlatCam design has many attractive properties besides
its slim profile. First, since it reduces the thickness of the camera
but not the area of the sensor, it can collect more light than a
miniature, lens-based camera of the same thickness. The light

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016