Signal Processing - September 2016 - 71

parameters include viewing angle, illumination angle, spatial
scale, spectral range, and polarization [39], [62]. The sampled
Algorithm
Robotic or
and
light may be emitted from a point, patch, object, or global
Computational
Dome Imaging
Application
scene. The plethora of variations for appearance specifications
Appearance
Recognition
Computational
leads to the basic question: What is the relevant definition of
Rendering
Global
Local
Imaging
appearance? The parameters are sometimes determined by the
Steganography
(Scene) (Surface)
capabilities of the imaging device, but the needs of the end-use
Diagnosis
Change Detection
application also drives appearance capture. Just as computaReflectance
3-D Modeling
tional imaging refers to the computation inherent in the imagMultiMultiMultiing process, we introduce the term computational appearance
Polarized
angle
scale spectral
to indicate how the working definition of appearance depends
on the computation in the subsequent algorithms and applicaFigure 1. Capturing appearance with imaging. Appearance comprises
tions. Figure 1 illustrates computational appearance as depenreflectance captured from the global scene or from a local surface patch;
dent on both the constraints of the imaging device and the
it may be multiangle, multiscale, multispectral, and include polarization
needs of the application algorithm.
information. Domes of lights and cameras or robotic manipulators can
For example, appearance is often used for recognition in
be used for appearance capture. Alternatively, computational imaging
computer vision. Reflected intensity from an object or scene
provides appearance using specialized devices that are typically faster and
more portable. Imaging provides appearance to the end-use algorithm or
is dependent on environment lighting, camera view, and surapplication. The bidirectional arrow indicates that the computation within
face geometry. The human visual system handles this variation
the end-use algorithm may determine the specifications of computational
and maps the set of possible appearances to a unique object
appearance and therefore affect the imaging process.
representation for recognition. The representation of an object
within the human visual system is invariant to environment
the color nor the shape are of interest in this recognition. Instead,
lighting and camera pose. For instance, a friend is recognizit is the specific way these surfaces reflect light that provide their
able at morning or at midday; recognition persists when the
identifying signature. Though the geometry of objects is not typiillumination direction has changed. Similarly, recognition is
cally controllable, illumination and camera pose can be manipunot interrupted when an observer moves and the viewpoint
lated in a strategic manner to capture comprehensive reflectance
changes. This geometric and photometric invariance is a prifor object and surface modeling. Figure 5 shows representative
mary goal in computer vision and invariant representations
cubes of different materials; neither the color nor geometry are
of appearance provide a mechanism to recognize objects
key in describing the object properties; the unique reflectance
and scenes. A strategy to develop these representations is to
properties of cube discriminate or render the materials.
observe the scene under variations in scene illumination and
Beyond basic object recognition, example applications for
camera pose (camera position and orientations). These comcomputational appearance are numerous and include e-comprehensive measurements are the input to modeling algorithms
merce, art archiving, robotic navigation, architecture, and biowhich provide a computational appearance descriptor as illusmedical applications such as dermatology. For e-commerce,
trated in Figure 2. For automated recognition, the appearance
conveying material appearance is an important component of
descriptors form a training set can be used to build a machinethe online presentation of consumer goods such as shoes, clothlearning classifier so that a class label is provided as the output.
ing, furniture, linens, and home decor. Color and geometry are
For computer graphics rendering, computational appearance is
typically understood as important in digital models, but mateused to render the complex appearance including effects such
rials enhance the digital presentation and convey subtle visual
as gloss, sheen, texture and translucency. An important question is how best to measure appearance,
i.e., how to sample the space of illumination and camera pose, to have a sufficiently detailed descriptor.
Appearance
Dictionary of
Reflectance
Descriptor
Consider the surfaces depicted in
Appearance
Figures 3 and 4. Figure 3 shows two
Descriptors
surface snapshots that fail to convey
Class
appearance in a meaningful manner;
Label
Reduce
i.e., an observer cannot identify the surMeasure
Dimensionality
Recognition
Appearance
face from either photo. Figure 4 depicts
Compute Invariant
material folds that reveal how the surRepresentation
face reflects light over variations in
the incident illumination and viewing
Figure 2. In an automated recognition pipeline, reflectance is measured and then represented by a
direction. In this manner, the materials lower-dimensional invariant appearance descriptor. A dictionary of such descriptors obtained from a
satin and velvet are easily discerned by training set is used to recognize the object using machine learning. The recognition result is a class
a human observer. Notice that neither label identifying the object, surface, or scene.
IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016