Signal Processing - September 2016 - 38

Dark
Current

Photon
Noise

PRNU
Factor

ADC

CRF

Final Pixel
values

Analog
SCENE
Gain
SHUTTER
SENSOR
Scene LENS
Irradiance
Exposure
Energy
Radiance d ω
∫ dp
∫ dτ
∫
∆p
t
A
W
J
J
W
m 2 sr Integration m 2 Integration m 2
Integration
Sensor
Over Pixel Saturation
Over Aperture
Over Time

IMAGE

Readout
Noise

Figure 3. The standard imaging pipeline in modern digital cameras, inspired by diagrams in [9] and [10]. The radiance from scene rays captured by the
camera are first integrated over the angle subtended by the lens aperture, over the time the shutter is open, and over the pixel's footprint area. This energy
can then be cut off by the saturation of the photon well at that pixel sensor, which limits the camera's dynamic range. The result is then quantized by an
ADC, and the CRF is applied to get the final digital pixel values. Different kinds of noise or error are injected at various stages in the pipeline, as described
in the article text. (Lighthouse image designed by Freepik.com.)

sensor called irradiance (E; units: W/m 2) . This irradiance is
then integrated over the time the shutter is open to produce an
energy density, commonly referred to as exposure (X; units:
J/m 2) . If the scene is static during this integration, the exposure can be written simply as X (p) = E (p) ·t, where p is the
point on the sensor and t is the length of the exposure (integration time).
The exposure can then be integrated over the pixel's footprint (integrating away the m2 term) to result in the total
energy (units: J ) accumulated in each pixel's photon well.
The measured energy is then read out by an analog-to-digital
converter (ADC), often with an analog gain factor applied
to amplify the energy before it is converted. For non-raw
images, the digital value is then mapped through a nonlinear
camera response function (CRF) to emulate the logarithmic
response of the human eye and make the final image look
better. This produces the final pixel values that are output in
the image file.
Two aspects of the pipeline limit the sensor's dynamic
range of measurable light. First, the pixels' photon wells are
of finite size and will saturate if too much energy is accumulated, creating an upper limit for the amount of light energy
that can be measured at each pixel. Second, the minimum
amount of detectable light is limited by the sources of noise
in the imaging pipeline. The first is dark current, which is
caused by thermal generation and induces a signal even if no
photons arrive at the sensor (i.e., it is dark). Next is photon shot
noise, which is caused by the discrete nature of light and is the
variance of the number of photons arriving at the sensor during exposure time t. Like many arrival processes, this count
is modeled by a Poisson random variable, the expected value
(as well as the variance) of which is based on the true irradiance E(p). The spatial nonuniformity of the sensor also causes
different pixels to respond differently to the same amount of
incident photons, which is modeled by the photo-response
nonuniformity (PRNU) factor. Finally, there is readout noise
caused by thermal generation of electrons when the signal is
being read from the sensor.
Given all of these noise sources (excepting dark current),
the actual measured exposure value Xt ( p) for well-exposed
38

regions can be modeled as a Gaussian random variable with
mean and variance [4]
n Xt (p) = ga ( p) E (p) ·t + n R
2

2

2

v Xt ( p) = g a ( p) E ( p) ·t + v R,

(1)

where g is the camera gain, a ( p) is the PRNU factor for the
pixel, and n R and v 2R are the readout mean and variance,
respectively. The Poisson nature of the photon shot noise is
responsible for the pixel variance's dependence on the irradiance. Without loss of generality, we can think of this measured exposure Xt ( p) at each point p in the sensor as being
mapped to a final digital pixel value Z ( p) with a function f
that effectively combines the CRF with the quantization and
saturation steps: Z ( p) = f (Xt ( p)) .
The challenge of HDR imaging, therefore, is to recover
the original HDR irradiance E(p) from noisy LDR images
such as Z ( p) . To do this, two main approaches have been
proposed: 1) specialized HDR camera systems that measure
a larger dynamic range directly and 2) capturing a stack of
differently exposed LDR images that are merged together to
produce an HDR result, as described in the following two sections, respectively.

Specialized HDR camera systems
Previous work on specialized HDR camera systems can be
divided into two main categories: 1) those that modify the
measurement properties of a single sensor to capture a larger
dynamic range and 2) those that use prisms, beamsplitters, or
mirrors in the optical path to image a number of sensors at
different exposures simultaneously.
In the first category, researchers have proposed HDR sensors that measure light in alternate ways, such as measuring
the pixel saturation time [11], counting the number of times
each pixel reaches a threshold charge level [12], or incorporating a logarithmic response like that of the human eye [13].
Others, such as Nayar and Mitsunaga [14], have proposed to
fit different neutral-density filters over individual pixels in
the sensor to vary the amount of light absorbed at each pixel.

IEEE SIgnal ProcESSIng MagazInE

|

September 2016

|
http://www.Freepik.com
Table of Contents for the Digital Edition of Signal Processing - September 2016
