Signal Processing - September 2016 - 41

reduce visible seams between different patches, the authors
apply Poisson blending to the final results.
In the second category are rejection algorithms without a
reference image, which must select a "static" subset of images
at every pixel to merge to produce HDR values. These methods
have a fundamental advantage over those that utilize a single
reference image because motion may occur in areas where the
reference might be poorly exposed. At these pixels, an HDR
value cannot be properly computed solely from the reference
image. However, rejection algorithms that do not use a reference must ensure that subsets are selected for neighboring pixels in a way that does not introduce artifacts.
Reinhard et al. [3] proposed one of the earliest methods
in this category. For every pixel that is deemed to be affected
by motion, the authors try to use the longest exposure that
is not saturated (effectively, a single-image subset). To deterAlgorithms that reject misaligned information
mine which pixels are affected by motion, they first compute
A second set of algorithms for HDR reconstruction assume
the variance of the irradiance values at each pixel p, weighted
that the camera is static (or that the images have been preregistered using a rigid alignment process, such as those
to exclude poorly exposed pixels. This estimated variance is
described in the "Algorithms That Align the Different Expothen thresholded, and the result is smeared out with a 3 # 3
sures" section) and that the scene motion is localized, meankernel to reduce edge and noise effects. Adjacent regions
ing that the majority of pixels contain no motion artifacts.
are then joined together to form the "ghosted" regions for
The basic goal of these methods is to idenwhich a single image from the stack will
tify those pixels that are affected by motion
be used. To select which image they will
The biggest problem with
and those that are not. The pixels that do
use for each region, the authors find the
rejection algorithms is
not contain motion artifacts can be merged
biggest irradiance value in the region that
that they cannot handle
using the standard HDR merging algois not in the top 2% (deemed to be outliers).
dynamic HDr content
rithms described in the "HDR Imaging
They then select the longest exposure that
Using Image Stacks" section. For the pixels
because they do not move includes this value within its valid range
that are affected by motion, however, only
to fill in this ghosted region, because the
information between
a subset of the images deemed to be static
longest exposure will contain least noise.
pixels but rather only
at these pixels will be merged to suppress
To further suppress artifacts, Reinhard et
merge information from
artifacts from moving objects.
al. linearly interpolate this exposure with
corresponding pixels
To accomplish this, two different kinds
the original HDR result, using the per-pixel
across the image stack.
of rejection methods are possible: 1) those
variance as a blending parameter.
in which a reference image is specified by
An alternative approach is proposed by
the user and 2) those that do not use a reference image. For
Khan et al. [31]; here, instead of detecting and handling difalgorithms in the first category, the user first selects an image
ferently the pixels affected by motion, the authors propose to
from the stack as the reference. These algorithms then simply
iteratively weight the contribution of each pixel depending on
revert back to this reference for any pixels where motion is
the probability of its being static (i.e., belonging to the backdetected so that the main difference between them is in how
ground of the scene). To do this, they assume that most of the
they detect motion. For example, the method of Grosch [29]
pixels are of the static background and so determine the probassumes two images in the stack and predicts values in the
ability of a pixel being static by measuring its similarity to the
second image by multiplying the values in the reference by the
neighborhood around it.
ratio of the exposure times, taking into account the nonlinear
Finally, some recent methods cleverly use rank minimicamera response curves. With this approach, a pixel is deemed
zation to deghost HDR images [32], [33]. These methods are
to be affected by motion if the actual color is beyond a given
based on the observation that if the scene is static, the different
threshold from the predicted value. In such cases, the algoexposure images X ( p) would simply be linear scalings of one
rithm simply reverts back to using the values in the reference
another. Therefore, they use the different exposure images to
image for these pixels.
construct a matrix and essentially minimize its rank to solve
Gallo et al. [30] improved on this work by using the logfor the motion-free image.
irradiance domain to do the threshold comparisons. Further,
The biggest problem with these and other rejection algofor robustness they compare patches instead of individual
rithms is that they cannot handle dynamic HDR content
pixels, so that a patch from an image in the stack would be
because they do not move information between pixels but rathmerged with the corresponding patch from the reference only
er only merge information from corresponding pixels across
if a certain number of pixels meet the threshold constraint. To
the image stack. Therefore, if different parts of a moving HDR
an energy-based optical flow optimization robust to changes in
exposure. Specifically, their energy function has a data term
that encourages the image to align to the reference and a regularizer that enforces smooth flow wherever the reference is
poorly exposed. However, these alignment algorithms all suffer from the problem of finding good correspondences, which
is extremely difficult, in particular for highly dynamic scenes
with deformable motion (e.g., a person moving). Furthermore,
scenes with occlusion and/or parallax do not even have valid
correspondences between the images in these regions, making it impossible to align the images in the stack correctly.
Therefore, the HDR results from alignment algorithms often
still contain objectionable ghosting artifacts for scenes with
complex motion.

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016