Signal Processing - September 2016 - 62

Real Sensor
Plane
q

Pixel

Virtual Sensor
Plane

∆u

Outside

Inside

Image of a
Pixel ∆ s

Physical Space

Figure 3. Light field imaging with a moving standard camera. Sensor
pixels in the sensor plane q are mapped outside the camera and inside
the world space. The camera lens and the image of the pixel constitute a
two-aperture pair, i.e., a unique phase space region. The color gradient in
the ray bundle indicates that the rays are considered to be virtual in the
camera's image space. In reality, the rays refract and are converged onto
the indicated pixel. In the world space, the ray bundle represents those
rays integrated by the pixel. The sensor has more such pixels (not shown
in the figure). These additional pixels effectively constitute a moving
aperture in the plane of the virtual sensor position.

pixel size and the pixel spacing, may not be correlated in real
applications, with corresponding implications for aliasing or
oversampling (see the "Computational Processing" section).
Going back to the physical meaning of these phase space
regions' respective ray bundles, we can conclude that each
pinhole/pixel combination yields a single measurement (i.e., a
single sample of the light field function) through integration
by the pixel. The phase space region, therefore, represents the
spatio-directional sampling kernel introduced by the finite size
of the pixel and the pinhole, respectively, while the center ray/
phase space point indicates the associated sampling position.
A key optical concept, the optical invariant, posits that
an ideal optical system does not change the volume of such a
phase space region (also known as étendue). As an example,
free-space transport, as a particularly simple propagation,
maintains phase space volume; it is described by a shear in the
horizontal direction of the phase space. Free-space transport to
a different plane is a necessary ingredient for computing refocused 2-D images from the light field.

Light field sampling with camera arrays/moving cameras
Obviously, pinhole images are of a low quality due to blurring
by the finite pinhole area-or, depending on its size, diffraction effects-and to the low light throughput. Introducing a
lens in the light field plane p improves the situation. This measure has the side effect of moving the apparent position of the
sensor plane q in front of the light field plane p if the sensor
is positioned at a farther distance than the focal length of the
lens, as shown in Figure 3. The ray bundles being integrated by
62

a single pixel can still be described by a two-aperture model
as before; however, at this point the model must be considered
virtual. This implies that it may intersect scene objects. It is
understood that the virtual aperture does not affect the scene
object in any way. The key point is that the refracted rays in the
image space of the lens can be ignored as a way of simplifying
the description. Only the ray bundles in the world space that
are being integrated by the pixel are considered.
With this change, the sampling of the light field remains the
same as before: instead of moving a pinhole, a moving standard
2-D camera performs the sampling task. Only the parameterization of the directional component s needs to be adapted to
the camera's intrinsic parameters. This is how pioneering work
was performed [2], [3]. Of course, this acquisition scheme can
be implemented in a hardware-parallel fashion by means of
camera arrays [8], [12].
Given a sampled light field l (u, v, s, t) and assuming full
information to be available, the slices I (s, t) = l (u = const.,
v = const., s, t) as well as I (u, v) = l (u, v, s = const., t = const.)
correspond to views into the scene. The function I(s, t) corresponds to a perspective view, while I(u, v) corresponds to an
orthogonal view of the inside space. These views are often
referred to as light field subviews.

Optics for light field cameras
While camera arrays can be miniaturized as demonstrated by
Pelican Imaging Corp. [12] and differently configured camera modules may be merged as proposed by LightCo. Inc. [13],
there are currently no products for end users, and building and
maintaining custom camera arrays is costly and cumbersome.
In contrast, the current generation of commercial light field
cameras by Lytro Inc. [14] and Raytrix GmbH [15] has been
built around in-camera light field imaging, i.e., light field imaging through a main lens. In addition, attempts are being made to
build light field lens converters [16] or use mask-based imaging
systems [17] that can turn standard single-lens reflex cameras into
light field devices. All devices for in-camera light field imaging
aim at sampling a light field plane p inside the camera housing.
To understand the properties of the in-camera light field and
their relation to the world space, we now extend the previous
discussion of general light field imaging to the in-camera space.

In-camera light fields
In-camera light fields allow the light field to be transformed
from the world space into the image space of a main lens,
where it is acquired by means of miniature versions of the
camera arrays, outlined earlier, that are most often implemented using micro-optics mounted on a single sensor. The
commercial implementations involve microlenses mounted
in different configurations in front of a standard 2-D sensor.
Each microlens with its underlying group of pixels forms an
in-camera (u, v, s, t) sampling scheme, as described in the
previous section. We may also think of these as tiny cameras,
with very few pixels, observing the in-camera light field. The
image of a single microlens on the sensor is often referred to
as a micro-image.

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016