Signal Processing - September 2016 - 68

A note on aliasing
It is commonly stated in the literature that an aliased acquisition is
required for superresolution [37]. In light of the previous discussion, we may make this statement more precise by stating that 1) a
Lambertian scene model is implied for geometric superresolution,
2) the samples should be jittered along the slope corresponding to a
scene point's depth, and 3) smaller phase space kernels associated
with the samples will be beneficial as long as there is still overlap
between them when propagated along the lines to construct the
superresolved subview. In conclusion, light field cameras may be
more suitable for implementing superresolution schemes than multicamera arrays due to their denser sampling of the phase space.

Image synthesis
Once the light field function is reconstructed, novel 2-D views
can be synthesized from the data. The simplest visualization is to
extract the light field subviews, i.e., images of constant (u, v) or (s,
t) coordinates, depending on the sampling pattern of the specific
hardware implementation. It should be noted that both choices, in
general, yield perspective views. This is because in-camera orthographic views [as synthesized by fixing the (s, t) coordinates] map
to a world-space center of projection in the focal plane of the main
lens. The subviews correspond to the geometry of the world-space
light field plane p w and the world-space virtual sensor plane q w
and, therefore, show a parallax between views. Interpolated subview synthesis has been shown to benefit from depth information
[40]: available depth information, even if coarse, enables aliasingfree view synthesis with fewer subviews.
The goal of light field image synthesis, however, is the creation
of images that appear as if they were taken by a lens system not
physically in place (see Figure 9). The example most commonly
shown is synthetic refocusing [29]. The technique, in its basic
form, consists of performing a free-space transport of the worldspace light field plane to the desired focus plane. After performing
this operation, an integral over the directional axis of the light field,
i.e., along the vertical dimension in our phase space diagrams,
yields a 2-D view focused at the selected plane. Choosing only
a subrange of the angular domain lets the user select an arbitrary
aperture setting, down to the physical depth-of-field present in the
light field subviews, that is determined by the sizes of the two (virtual) apertures involved in the image formation. If spatio-directional superresolution techniques (as described in the "Computational
Processing" section) are employed, this limit may be surpassed.
Computing the 4-D integral allows for general settings: even
curved focal planes are possible by selecting the proper phase
space subregions to be integrated. However, it can be computationally expensive. If the desired synthetic focal plane is parallel to the world-space light field plane p w and the angular
integration domain is not restricted, Fourier techniques can yield
significant speedups [14]. If hardware-accelerated rendering is
available, techniques based on texture-mapped depth maps can
be efficient alternatives [16].

Conclusions
With almost a quarter century of practical feasibility, light field
imaging is alive and well, gaining popularity and progressing
68

into the market with several actors pushing for prime time.
There are still sufficiently many scientific challenges to keep
researchers occupied for some time to come. In particular,
the bar of resolution loss must still be lowered in the hope
of increased consumer acceptance. The megapixel race has
slowed down, and pixel sizes are approaching their physical
limits. This implies larger sensors and, thus, increased expense
for additional resolution increases that would benefit light field
technology. Improved algorithmic solutions are, therefore, of
fundamental importance.
The next big step will be light field video, pushing optical
flow toward scene flow and associated projected applications,
such as automatic focus pulling, foreground/background segmentation, space-time filtering, etc. In terms of applications, we
are seeing 4-D light field ideas penetrating in both the small
and the large. In the small, we are seeing the emergence of light
field microscopy [41], although we need improved aberration
models and, eventually, expanded wave-optical treatments [39].
In the large, sensor networks will become increasingly important. More complex scenes-such as translucent objects [42] or,
more generally, non-Lambertian scenes [43]-are made possible. Crossover to other fields, such as physics, are also appearing
[44]. These are surely exciting times as we head into the second
quarter century of light field technology.

Acknowledgments
We would like to acknowledge the work of all light field
researchers, in particular the work of those whom space constraints have prevented us from citing. You are tackling the confusions of 4-D, slowly but steadily creating the basis for a new
understanding of imaging technology. Special thanks go to Jan
Kucera for developing and sharing the Lytro Compatible Viewer
and Library as well as to Donald Dansereau for the development
of the MATLAB Light Field Toolbox. This work was supported
by the German Research Foundation through Emmy-Noether
fellowship IH 114/1-1 and the ANR ISAR project.

Authors
Ivo Ihrke (ivo.ihrke@inria.fr) is currently a researcher with Carl
Zeiss, Inc. From 2013 to 2016, he was a permanent researcher at
the Inria Bordeaux Sud-Ouest Research Institute, France; previously, he was an Emmy-Noether fellow with the German
Research Foundation, and he headed a research group within the
Cluster of Excellence, Multimodal Computing and Interaction, at
Saarland University, Germany, which was also associated with the
MPI Informatik and the Max-Planck Center for Visual Computing
and Communications (2010-2012). He was a Humboldt fellow at
the University of British Columbia, Vancouver, Canada (2008-
2009). He received his M.S. degree in scientific computing from
the Royal Institute of Technology, Stockholm, Sweden, in 2002
and his Ph.D. degree (summa cum laude) in computer science
from Saarland University in 2007. He coorganized a workshop on
computational cameras and displays at the 2012 Conference on
Computer Vision and Pattern Recognition, a Dagstuhl seminar on
computational imaging in 2015, and the Zeiss Symposium
Workshop on Computational Imaging in 2016.

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016