Signal Processing - November 2016 - 41

PERCLOS Blinking

expressions such as seating posture, speaking, and head gesfor fatigue detection focus on the eye area, using detailed
tures have been studied in the literature to infer human states
geometric measurements of iris, pupils, eyelids, and some[24], [28]. For example, activities related to distracted driving
times the specular point on the iris. These measures are also
such as answering a cell phone, eating, and smoking can be
used for gaze estimation, in which head pose estimation is
detected by hand tracking [29]. Moreover,
also necessary. In general, there are two
vision systems looking at feet and hand
approaches for head movement tracking:
Today's cameras are
positions can directly monitor driver's interthe rigid three-dimensional (3-D) head
empowered with abilities
action with the vehicle, and use this informodel [24] and front face orientation
beyond human vision,
mation for maneuver prediction [30].
estimation using facial landmarks [25].
and thus can capture
It is worth mentionin that monitoring a
Metafeatures can also be calculated based
extremely subtle changes, driver's facial and body expressions also
on statistics of the aforementioned visual
such as HR using
provides possible means for determining
cues, for example frequency, average inthe driver's emotions. These measures are
tensity, or standard deviation in a certain
thermal imaging.
successfully used in other contexts such as
time window. Using multiple metafeatures
human-computer interaction studies for affect recognition.
often improves the system performance, where a fuzzy inferThere is great potential to use facial expressions for affect
ence system [22] or Bayesian network [25] can be used for
recognition in the context of driver monitoring, which has not
data fusion. Detection can also be performed by learning spebeen explored yet. Finally, it is noteworthy that facial/body
cial patterns from the temporal signals using algorithms such
expressions do not necessarily need to be measured using
as support vector machines [26]. Data fusion and inference
video recorders. For example, eye movement, gaze direction,
methods will be further discussed the "Data Fusion" section.
and blinks can all be monitored by attaching a set of electriAn alternative approach is the use of holistic feature extraccal sensors to the face around the eyes and measuring the
tion algorithms that do not require any handcrafted features
electrooculogram (EOG) signals. However, this method is
and extract statistically discriminant features/patterns from
highly invasive and is not suitable for realistic driving condithe image frame using holistic machine-learning and pattern
tions. Another example of monitoring body expression withrecognition techniques. This approach is widely used in other
out video recorders is the use of sensors embedded in the
areas (e.g., face recognition, emotion recognition) and has
driver's seat to measure the body pressure distribution on the
been shown to have great potential; however, it has not yet
seat, from which certain body postures can be detected [11].
been implemented in driver monitoring.
Today's cameras are empowered with abilities beyond
human vision, and thus can capture extremely subtle changes,
Physiological measures
such as HR using thermal imaging [27]. These technologies
Physiological measures can recognize the changes in driver's
potentially enable nonintrusive evaluation of driver physiolarousal level [31]. Arousal is largely affected by the autonoogy. Another example is the use of pupillary dilation as an
mous nervous system (ANS), which includes sympathetic
indicator of cognitive load. In addition to facial features, body
and parasympathetic branches. The sympathetic branch

(a)

Image
Acquisition

Yes
Other Information
No
1
0.5
0
0

(b)

Detection
and
Tracking

Time of the
Day
5
10
Time (Seconds)
(c)

Meta-Feature
Extraction

Classification
and Inference

Data
Fusion

Feature Extraction

Vehicle
Measures

Driver's
Vigilance Level

Decision Making

FIGURE 5. An example of driver monitoring pipeline using facial and body expression measures, depicting typical processing steps and intermediate outputs from them: (a) video frames acquired, (b) tracking of driver's face (yellow rectangular box) and facial landmarks (green dots), and (c) example time
series plot of two metafeatures: blinking (binary variable) and PERCLOS (scales from 0 to 1 representing eyes open to close).
IEEE SIgnal ProcESSIng MagazInE

November 2016

Table of Contents for the Digital Edition of Signal Processing - November 2016