Signal Processing - March 2017 - 85

BMW 7 Series cars have gesture control
to more accurate tracking results, but also
Other types of displays,
for some features) is driving the developoffers vital information on intent, revealsuch as head-up displays
ment of automotive-grade gesture tracking potential conflict or threat [16], [21],
and general threeers [15]. In Figure 1, a tracker provides, in
[22]. Destination prediction can be viewed
dimensional displays, have real time, the pointing hand/finger(s) locaas a means to assist planning and decision
the potential of providing
tions, Y1:k _ " Y1, Y2, ..., Yk , , at the discrete
making at a system level higher than that of
established conventional sensor-level tracktime
instants t 1, t 2, ...., t k . For instance,
a more immersive driving
ing algorithms, whose objective is to infer
Yn = 6xt t n yt t n zt t n@T is the 3-D Cartesian
experience and are
the current value of the latent state X t (e.g.,
coordinates
of the pointing fingertip at tn.
becoming increasingly
the tracked object position, velocity, etc.)
In general, the predictive display demands
commonplace in vehicles. reliable pointing finger tracking at a rate
[22]. For example, destination-aware trackers that include an additional mechanism
exceeding 30 Hz, as the majority of in-vehito determine the object end point are proposed in [16]. These
cle pointing tasks can have durations in the range of
methods discretize the state space area into predefined regions,
0.2 s G T G 4 s [8]. Figure 3 depicts three complete 3-D
and the object can only pass through a finite number of these
pointing trajectories, Y1:T , collected in a car using an LM
zones; such a discretization can be a burdensome task for freecontroller under three conditions, which visibly affect the
hand pointing gestures in 3-D. On the contrary, the predictive
pointing gesture.
display solution presented in this article uses continuous state
space motion models that do not impose any restrictions on
Bayesian intent inference
the path the pointing finger has to follow to reach its intended
Let D = " D i : i = 1, 2, ... , N , be the set of N selectable
on-display end point and can easily handle noisy as well as
items on the interactive display. While no assumptions are
asynchronous observations. Nevertheless, other conforming
made about the layout of the icons in D, each item is moddestination-aware tracking methods can be applicable.
eled as a distribution representing the extended regions in
A related scenario in which there is a growing interest is
space of various shapes and sizes occupied by the correthe user input on a smartphone, perturbed due to situational
sponding GUI elements. For simplicity and computational effiimpairment, for example, walking [17]. Typically in such
ciency, Gaussian distributions can be considered, and the ith
cases, the GUI is dynamically adapted to compensate for the
item is modeled as N (n i, C i) . The mean n i and covariance
measured noise. For an in-vehicle display, the pointing time
matrix C i capture the 3-D location and the extent-orientation
and distance are notably longer than that for a handheld device
of D i, respectively. At the time instant t k , the inference modand the correlation between the pointing hand movements
ule in Figure 1 calculates the posterior probabilities for the
and the experienced in-car accelerations or vibrations can be
N destinations,
ambiguous [10]. This is attributed to the complexity of the
human motor system and its response to noise as well as the
P (t k) = " p (D I = D i Y1: k) : i = 1, 2, ..., N ,,
(2)
seat position, cushioning, reaching style or distance, etc. Thus,
compensating for the measured in-vehicle noise can have limited effects on improving the display usability. Here, perturbed
user input is tackled within the statistical inference framework
of a predictive display.

An in-vehicle predictive display system

200
Y (mm)

175

Gesture tracker
Motivated by extending HCI beyond traditional keyboard input
and mouse pointing, new 3-D vision sensory devices have
emerged that can track, at high rates, hand gestures, including
pointing fingertip(s), e.g., Microsoft Kinect, Leap Motion
(LM), and SoftKinetic DepthSense. However, operating in a
mobile vehicle enviroment can be challenging to these trackers
due to dynamically changing light conditions, in-car vibrationsaccelerations, occlusion with limited in-car mounting positions, large coverage area (e.g., steering wheel or armrest to
display and the front passenger), and others. Fortunately,
the current interest in gesture-based HCI in cars (e.g., current

Stationary Car
Motorway Driving
Off-Road Driving

225

Next, we describe the various modules that compliment the
present in-vehicle interactive display, e.g., a touch screen, to
realize the intelligent predictive display system in Figure 1.

150
125

100

t1
t1
t1

140
50
105
00 0
1
- -8 60
60
- 40 0
15
- 2 0
X
(mm)
-
-30
20 0
Z (mm)
4 0 -75
6

Figure 3. Recorded 3-D pointing fingertip tracks to select icons (circles)
on an in-car touch screen (blue surface) under various driving/road
conditions [10]; t 1 1 t k .

IEEE SIgnal ProcESSIng MagazInE

March 2017

Table of Contents for the Digital Edition of Signal Processing - March 2017