Signal Processing - September 2016 - 144

(a)

(b)

(c)

Figure 3. Judder cancelation: the original 60-Hz input video is first upsampled to 3,000 Hz, then filtered and downsampled again to 60 Hz to simulate different exposure times. (a) A frame from the original sequence. (b) Conventional long-exposure simulation by blending multiple interpolated frames; note
that the entire image is blurred due to camera motion. (c) Gaze-adaptive filtering keeps the foreground sharp while the background is correctly blurred,
consistent with smooth-pursuit eye movement of the foreground. (Original footage copyright © 2013 RED Digital Cinema. All rights reserved.).

where t is time, x is a screen coordinate, and S (x + p (t), t) is
irradiance received by the retina. This corresponds to translating each frame in the opposite gaze path direction, compensating eye motion, and setting the spatial components
along the path to zero: by integrating along the temporal axis
and translating the frames back to their original position, the
frames are filtered according to the gaze path. This filter
operation may be applied in real time for adjusting the integration time if the temporally upsampled video frames can be
accessed quickly enough. The result is a consistently filtered,
temporally downsampled video sequence that can be displayed on any conventional display or projector (Figure 3).
Besides correct prefiltering of low as well as high frame-rate
video, the approach can also be applied to simulate virtually
any shutter configuration. Rolling shutter, focal plane shutter, or
artistic shutter effects, e.g., shutter times longer than one frame,
can be created this way from the same footage during postprocessing. Conversely, still images can be infused with localized
motion blur to convey information about scene motion.

Boosting perceived resolution
Big-size screens featuring 60+ in diagonals and 4k pixels per
line are entering the consumer market. HMDs that are the
driving force behind virtual reality applications, on the other
hand, must be small in size but at the same time cover a very
wide field of view. Even with state-of-the-art 350 PPI screens,
at a natural field of view current HMD displays are still an
order of magnitude away from eye acuity (Table 1), exacerbated even more by significant pixel magnification in the central
region of lens-based HMDs.
Recently, apparent-display-resolution enhancement techniques have been proposed to provide super-resolution on high
refresh-rate displays [26]. By exploiting how the HVS observes
144

and processes moving content, these approaches are able to
boost perceived resolution beyond the actual physical resolution
of the display. With active-matrix organic light-emitting diode
technology, refresh rates in excess of 1,000 Hz are achievable,
allowing for a six-fold increase in apparent display resolution. In
essence, apparent-display-resolution enhancement allows trading screen refresh rate for perceived resolution as long as the
user's gaze continuously and predictably tracks moving foreground via smooth-pursuit eye movement.
Didyk et al. have shown that retinal integration of highframe-rate, low-resolution images results in an increase in
perceived resolution if the frames are displayed above the critical flicker frequency [26]. To perceive an object at higher than
screen resolution, the eyes must smoothly pursue the object
while it is moving across the screen. Smooth-pursuit eye movement can already be evoked by simply moving a static image at
constant velocity. In videos, the effect is achieved by exploiting
the movement of foreground objects [27]. Apparent-displayresolution enhancement is governed by two constraints: high
display-refresh rates are necessary to achieve high contrast
and strong resolution enhancement, and the motion must be
predominantly in a diagonal direction to achieve resolution
enhancement both horizontally and vertically. The more foreground object motion deviates from the diagonal direction on
the pixel grid, the less pronounced the resolution-enhancement
effect is in the direction orthogonal to the motion.
Apparent-display-resolution enhancement enables experiencing a standard low-frame-rate, high-resolution video at full
resolution on a much lower resolution, yet high refresh-rate
display [25]. Based on the assumption that our gaze follows
the most salient regions of the sequence, the salient foreground
regions are determined in a preprocess. The video frames are
then continuously, unnoticeably shifted in such a way that, in

IEEE SIgnal ProcESSIng MagazInE

September 2016

Table of Contents for the Digital Edition of Signal Processing - September 2016