Signal Processing - September 2016 - 143

IEEE SIgnal ProcESSIng MagazInE

|

September 2016

|

Four Frames

(j)

Our Approach
Retinal Image

(h)
Ghosting
Two Frames

(f)

Four Frames

Retinal Image

Retinal Image
Holdtype Blur

Two Frames

(d)

Retinal Image
Holdtype Blur

(e)
(c)

(b)

(a)

Expected
Retinal Image

Eye Motion

(1)

taken with a static camera record the object as horizontal streaks. (d) When these streaks are displayed on a screen, the viewer perceives them as rectangles. (e) Shorter exposure times at fixed-frame rate lead to temporal
gaps. (f) On screen, the object is then perceived as a number of discrete, disconnected rectangles, causing perceptually annoying judder. (g) Higher frame rates alleviate the problem but (h) do not eliminate it. By taking
smooth-pursuit eye movement into account, (i) the video frames can be optimized along the known gaze path prior to display to achieve (j) a continuous, authentic motion perception [2].

S (x + p (t), t) dt,

ti

Figure 2. Temporal aliasing. (a) An object moves horizontally while eye gaze moves upward. (b) In the physical world, the perceived motion of the object is a continuous, diagonal movement. (c) Long-exposed frames

#

(g)
Eye Motion

Frame 0 Frame 1
Frame 0 Frame 1
Eye Motion
t1 - t0 = 2 × CFF

t1
Image Plane
t0

t i + TR

R i (x): =

Holdtype Blur

Frame 0
Frame 1
Frame 2
Eye Motion
Frame 3
(i)
Frame 0 Frame 2

Frame 1 Frame 3

Gaze-Based
Subframe
Integration

toward incorrectly sharp and detailed image regions. The
discontinous motion of the background distracts our visual
attention from the tracked foreground because our peripheral
vision is especially sensitive to movements and expects consistent, smooth motion (Figure 2).
To remedy the effects of temporal aliasing, Nvidia recently
released its G-Sync technique, which adapts display-refresh
rate to the processing time of each rendered frame (www.
geforce.com/hardware/technology/g-sync). The technique reduces some stutter motion artifacts by avoiding repeated display
of the same frame, allowing for smoother motion perception
in games.
Similar to judder, hold-type blur is another form of aliasing
that occurs when objects move faster in image space than they
can be captured by the camera. Motivated by the Nyquist theorem, object velocities above one pixel per frame in the image
space introduce aliasing. As a result, at 24-Hz capture frame
rate and HD resolution, the sampling criterion is violated
already for slow-moving objects. Higher frame-rate videos
with a typical frame rate of 48 or 60 Hz reduce recorded
motion blur and hold-type blur, yielding sharper perceived
images. High frame-rate movie cameras are currently being
explored by movie directors (e.g., in Peter Jackson's The Hobbit series), and specialized upsampling hardware is being
integrated into standard TV sets. High frame-rate video,
however, is not always practical or beneficial. The video
must be recorded at shorter exposure times and because there
is always some dead time to store each frame (or to open
the shutter), only about 60% of the time interval between
successive frames is being captured. Again, temporal aliasing may be the result. Some viewers even report perceiving
a distracting speedup of the video (www.pcmag.com/article2/
0,2817,2379206,00.asp).
Even at 60-Hz capture frame rate, fast-moving objects cannot be recorded free of aliasing. If eye movement is known
or can be reliably predicted-e.g., via passive, saliency-based
gaze prediction-the minimal sampling rate (or maximal
motion) between two frames can be estimated. Specialized
displays that support multiflash protocols are able to reduce
artifact visibility by showing each video frame multiple times
[23]. For conventional displays, however, only appropriate
frame prefiltering is able to yield convincing results [2]. To
this end, the original video sequence must first be temporally
upsampled. For frame interpolation, robust optical-flow-based
algorithms may be used [24]. Alternatively, perceptually
smooth results may also be obtained by directly making use
of predicted gaze [2], [25]. In both cases and for best results,
exposure time of the input sequence should be low to minimize
motion blur.
When we know the eyes' gaze path p during smooth-pursuit
eye movement, the correct retinal image R can be computed for
any point in time and any desired integration time TR by integrating the upsampled video sequence along the path

143
http://www.geforce.com/hardware/technology/g-sync http://www.pcmag.com/article2/
Table of Contents for the Digital Edition of Signal Processing - September 2016
