Signal Processing - September 2016 - 63

Unfortunately, the in-camera light field is a distorted version of the world coordinate light field due to refraction by the
main lens. Here, we encounter a classic misconception: mapping the world space into the image space of the main lens,
even by means of a simple thin-lens transformation, does not
result in a uniformly scaled version of the world space. Instead,
the in-camera light field is a projectively distorted version of
the world-space light field (see Figure 4), which results from
the depth-dependent magnification of optical systems.
There are different ways to describe this distortion, e.g., in
terms of phase space coordinates, as suggested by Dansereau et
al. [18], corresponding to a ray-remapping scheme or by appropriate projection matrices. The projection matrices commonly
used in computer vision to model camera intrinsics and extrinsics are not directly usable because they model a projection onto
the image plane of a 2-D camera. It is, however, important that
3-D information is preserved. The closest model is the OpenGL
projection matrices used in computer graphics to transform a
Euclidean world space into a space of so-called "normalized
device coordinates." This space is also a 3-D space, but a perspectively distorted one.

Interpreting in-camera light field imaging
in the world space

the right distance from the microlens array. Second, because a
microlens is often a one-lens system, its focal length is strongly
dependent on the wavelength of the light. The configuration
may be set for green light, but the red and blue wavelengths
are then focused at different distances. The finite pitch of the
pixels, however, makes the system tolerant to these issues.
In microlens-based light field imaging, the microlens plane
takes the role of the in-camera light field plane p. The virtual
sensor plane (i.e., the sensor plane transformed by the microlens array) takes the role of the second aperture, as in Figure 3.
The inverse action of the main lens, then, is to map these
two planes into the world space. In conjunction, they define
the properties of the light field subviews such as focal plane,
depth of field, viewing direction and angle, field of view,
and-through these parameters-the sampling pattern for
the world-space light field. Optically refocusing the main
lens (i.e., changing its position with respect to the microlens
array) affects most of these properties. The precise knowledge of the optical configuration is, therefore, necessary for
advanced image processing tasks such as superresolution, and
corresponding calibration schemes have been developed, as
discussed in the "Calibration and Preprocessing" section.

Optical considerations for the main lens

Thinking about how a miniature camera array is imaging the
The main optical considerations concern the (image-side)
distorted in-camera light field is a bit difficult. It is, however,
f-number of the main lens and the (object-side) f-number of the
possible to apply the inverse perspective transformation to the
microlenses, respectively. The f-number of an imaging system is
light field plane and the virtual sensor plane-i.e., to the two
the ratio of its focal length and the diameter of its entrance pupil.
aperture planes characterizing a light field sampling device-
It describes the solid angle of light rays that are passed by an
to obtain a world-space description in
terms of an equivalent camera array.
The detailed position of these two
In-Camera
World Space
Light Field
planes depends on the configuration of
Light Field
Light Field
Equivalent
the light field camera. There are essenCamera
tially two choices:
Array
■ an afocal configuration of the lens(Virtual)
lets [19]
Camera
■ a focused configuration of the lensMain Lens
lets [20], [15].
In the first case, the sensor plane is
positioned exactly at the focal distance
of the microlens array. In the second,
Microthere are two possibilities for creating
Camera
Array
real or virtual imaging configurations
(Real)
of the microcameras: by positioning
q p
q′
the sensor plane farther from or closer
pw
q w′
to the microlens focal length, respecVirtual
Sensor Light
Light Field
Virtual
tively. This choice has the effect of
Plane Field
Sensor
Plane
Sensor
Plane
(World)
Plane
Plane
placing the in-camera virtual sensor
(In-Camera)
(World)
plane at different positions: namely at
infinity for an afocal configuration or
in the front or in the back of the micro- Figure 4. The main lens images its object space (right) into its image space (left), distorting it in the
process. The world-space light field is, therefore, distorted into an in-camera light field. The distorlens plane for a focused configuration.
tion is a perspective projection, with its center at the center of the main lens's image-space principal
In practice, the first can only be plane. A micro-optics implementation of a camera array observes the distorted in-camera light field. An
approximately achieved. First, it is dif- equivalent camera array in world coordinates can be found by mapping the light field plane p and the
ficult to mechanically set the sensor at virtual sensor plane q to the world space.
IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016