Signal Processing - September 2016 - 43

sequence, while rejecting the pixels that are still misaligned,
they do not have valid correspondences in these regions and so
to avoid artifacts.
the images cannot be aligned in these parts. Patch-based HDR
The state of the art in HDR video reconstruction is the
reconstruction, on the other hand, is related to patch-based
work of Kalantari et al. [5], which extended the patchimage synthesis methods (e.g., for single-image hole filling)
based optimization work of Sen et al. [1] to produce coherbecause they both use a patch-based similiarity optimization
ent HDR video streams. Specifically, they modify the HDR
to resynthesize content in the final reconstruction without
image synthesis equation (3) to enforce temporal coherence
an underlying correspondence. Because of this advantage,
by performing a bidirectional similarity between adjacent
these methods have proved to be the most successful HDR
frames. In addition, they use optical flow
deghosting algorithms proposed to date.
during the optimization to constrain the
For example, a recent state-of-the-art
Patch-based HDr
patch-based search, which produces a
report by Tursun et al. [6] testing many
reconstruction is related
stream of high-quality HDR frames.
deghosting algorithms found that the algorithm of Sen et al. [1] and the later, related
to patch-based image
method of Hu et al. [34] ranked first and
Open problems and challenges
synthesis methods (e.g.,
second over other deghosting techniques by
Despite the tremendous progress of the
for single-image hole
a fairly large margin. The success of patchcomputational photography community
filling) because they
based optimization for HDR reconstruction
on HDR imaging over the last 20 years,
both use a patch-based
has led others to explore ways to further
many challenges remain. For example,
similiarity optimization to
improve the quality of these approaches. For
the capture of high-quality HDR images
example, Aguerrebere et al. [24] focused on
of highly dynamic scenes with convenresynthesize content in
reducing the noise of the estimated irraditional digital cameras is still a challengthe final reconstruction
ance. First, this method synthesizes a "refering problem. Although state-of-the-art
without an underlying
ence" containing well-exposed, de-ghosted
deghosting algorithms like the patchcorrespondence.
information in all parts of the image using
based optimization of Sen et al. [1] can
Poisson image editing (although the methsuppress many of the ghosting artifacts
od in Sen et al. [1] could also be used). Noise is then reduced
that would normally occur in these scenes, these methods
through a patch-based denoising method that finds all patches
cannot recover scene content that is poorly exposed in the
in the image stack within a threshold to each patch in the referreference image and is not visible in any of the other images
ence, where the L2 distance between patches is normalized by
in the stack. Moreover, the patch-based optimization in
these algorithms is computationally expensive and can take
the variance from (1). The MLE of the patch-centers at each
several minutes to compute an image. This limits the applipixel is then computed to significantly reduce the noise in the
cability of these methods to long video sequences or for
final result.
real-time, on-board computation in current smart phones,
for example.
HDR video
It is entirely possible that new sensor technologies, such
Up to now, we have focused exclusively on the HDR
as Fuji Film's recent Super CCD EXR sensor, will bypass the
acquisition of still images. However, the problem of capproblems inherent in stack-based methods by capturing a sinturing HDR video sequences is of considerable interest as
gle image with extended dynamic range. However, even these
well. For example, filmmaking companies incur a signifinew technologies will likely raise interesting questions, such as
cant cost to light sets, a cost that would be largely elimihow users will employ and interact with HDR images. Furthernated by high-quality, HDR video systems. For this
more, as HDR imaging becomes more mainstream, we expect
reason, professional movie camera system suppliers such
that new applications for HDR imaging (such as for medical
as RED have been pushing the dynamic range of standard
imaging or manufacturing) will be proposed and explored.
sensors. Moreover, specialized HDR camera systems such
as that of Tocci et al. [20] have been proved capable of
capturing high-quality HDR video, although they are not
Conclusions
yet widely available.
In this article, we first summarized the main aspects of HDR
For conventional digital cameras, the only way to capimaging, starting with an overview of the problem of limited
ture HDR video is to alternate exposures through the entire
dynamic range in standard digital cameras and the physical consequence. This problem was first tackled by Kang et al. [35],
straints responsible for this limitation. We then surveyed statewho use gradient-based optical flow to compute a bidirectional
of-the-art approaches developed to tackle the HDR imaging
flow from the current frame to neighboring frames and unidiproblem, focusing on both specialized HDR camera systems
rectional flows from neighboring frames to the current frame
and stack-based approaches captured with standard cameras.
(four flows total). Once computed, the flows can be used to
For the latter, we discussed algorithms to address ghosting artiproduce four warped images by deforming each of the two
facts that can occur when capturing dynamic scenes. Finally, we
neighboring frames. The resulting images can be merged with
discussed algorithms for capturing HDR video and concluded
the reference to produce an HDR image at every frame of the
with a review of open problems in HDR imaging. We hope that
IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016