Signal Processing - September 2016 - 65

v
u
t
s

u
Bayer Pattern

White Image

Subview Calibration

(a)

(b)

(c)

Figure 6. Light field preprocessing and calibration for a Lytro camera. (a) Using a Bayer pattern within the micro-images causes a shift of light field view
for the color channels because different colors sample different (s, t ) coordinates. (b) A white image (luminance) used for vignetting compensation, (c)
A subpixel determination of the centers of the micro-images enables a calibrated (s, t )-coordinate system to be assigned to each micro-image. The (u, v)
coordinates are sampled in a hexagonal fashion by the microlenses. The orientation of this global coordinate system also determines the rotation angle of
the (s, t ) system. The inset shows s and u calibration maps for the raw image.

the main lens. Commercial cameras, therefore, store significant amounts of calibration information in the internal camera
memory. As an example, the combined vignetting of a main
lens and microlenses changes across the field of view and with
the focus and zoom settings of the main lens. Therefore, white
images have to be taken for a sufficiently dense set of parameter settings. The closest white image to the parameters of a user
shot are then used for compensation. In a lab setting, it is advisable to take one's own white images prior to data acquisition.

Calibration
To properly decode the four light field dimensions from the
2-D sensor image, it is necessary to carefully calibrate the
(u, v, s, t) coordinates of every pixel that has been recorded by
the sensor. With current lenslet-based architectures, to the first
order this amounts to determining the center positions of the
lenslets and the layout of the lenslet grid [Figure 6(c)]. More
accurately, the position of the central view is given by the sensor intersection of the chief rays passing through the main lens
and each one of the lenslets. In addition, microlens aberrations
and angularly variable pixel responses can shift this position
[26]. In general, the responses are also wavelength dependent.
The lenslet grid is typically chosen to be hexagonal so as to
increase the sensor coverage. The spherical shape of the microimages and their radius are determined by the vignetting of the
main lens, which is the result of its aperture size and shape. The
tight packing of the micro-images is achieved by f-number matching, as discussed in the "Optics for Light Field Cameras" section.
It should also be noted that manufacturing a homogeneous lenslet
array is difficult and so some variation may be expected. Further, the mounting of the lenslet array directly on the sensor may
induce a variable distance between the sensor and the lenslets.
The calibration described here usually pertains to the in-camera light field coordinates. When assuming thin-lens optics for
the main lens, these correspond to a linear transformation of the

light field coordinates in the object space. Calibration approaches to determine this mapping are described by Dansereau et al.
[18] for afocal light field cameras; the techniques, as well as the
preprocessing steps described earlier, are implemented in their
Matlab Light Field Toolbox. Bok et al. [27] present an alternative
for performing a similar calibration by directly detecting line
features of a calibration target from the raw light field images.
Johannsen et al. [23] describe the calibration scheme for focused
light field cameras. The handling of the effects of optical aberrations by the main lens is usually performed using classical radial
distortion models from the computer vision literature. While
these measures improve the accuracy, they are not completely
satisfactory because the light field subviews suffer from nonradial distortions [see Figure 5(b)]. Lytro provides access to calibration information, including aberration modeling, through its
software development kit. Alternatively, modelless per-ray calibrations [28] using structured light measurements have shown
promising performance improvements. However, the need for a
principled distortion model remains.
Once a per-pixel calibration is known, the suitably preprocessed radiance values of the light field function can be
assigned to a sample position in the phase space. In principle,
reconstructing the full light field function amounts to a signal processing task: given a set of irregular samples in the
phase space, reconstruct the light field function on that space.
In practice, additional constraints apply and are used to, e.g.,
achieve superresolution or to extract depth. A prerequisite for
superresolution is having a known shape for the phase space
sampling kernels, also called ray-spread functions. Calibration schemes for these still have to be developed.

Computational processing
The reconstruction of the 4-D light field function from its
samples can be achieved by standard interpolation schemes
[2], [29]. However, the light field function possesses additional

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016