Signal Processing - September 2017 - 189

PERSPECTIVES

(continued from page 196)

measures as basic as relative speaking
times. Additionally, people are not
consistent in their judgments over time
due, in part, to changes in mood, focus,
and fatigue as well as learning effects. Finally, humans can only see and hear what
is observable and thus cannot directly
measure another's physiology (although
in many day-to-day interactions, this is
often for the best).
Signal processing is primed to have
a transformative impact on behavioral
science, a data-rich domain comprising
noisy signal data that holds information
on individuals' hidden mental states
and traits. Clinical experts will always
be critical to mental health assessment
and treatment, but computational methods can now support their efforts with
time-continuous, objective measures of
a social scene. In particular, since experts are unable to constantly observe
their patients and find quantification of
behavior challenging, behavioral signal
processing (BSP) methods can augment
their abilities [10]. BSP seeks to quantify qualitatively characterized behavioral constructs at scale using low-level
behavioral measurements.
Formally, the problem that we present
is that of identifying the hidden attributes
of the system that modulates the body's
signals, uncovered through novel signal
processing and machine learning on

True Behavior
True Mental
States

Neurological
Activations

Emotion
Intent

Speech and
Language

Condition
Beliefs
Knowledge

Why now?
The data ecosystem continues to grow
and become more integrated into our
daily lives, enabling multiple opportunities to positively affect human health

Sensing
fMRI
EEG
Audio
Video

Signal Processing
Signal Filtering,
Person Detection,
and Joint Modeling
of Channels and
Persons

ECG
Temperature
EDA

Interaction
Measures
Brain
Functioning
Estimators

Face and Body
Gestures

Environmental
Signals and
Metadata

Physiological
Measures
Y

Machine
Learning

Predicted
Psychological
Constructs
Emotion
Intent
Condition
Social
Ability
Empathy
Q

Psychological
Response

Behavioral
Informatics

Speech and
Language

Accelerometry
Face and Body
Gestures

and well-being. Low-cost physiological wearables that can derive direct measures of a person's internal state are
increasingly common as are wearable
vision and audio devices. The medical
Internet of Things (mIoT) market has
been projected to be worth US$117 billion by 2020 [11], representing 40% of
the total IoT market, but emerging signal processing research could push that
value even higher. Signal processing
has become the critical missing link in
translating ubiquitous sensors into real
impact on mental and behavioral health.
There is an immense need for extracting actionable information from multimodal biobehavioral signals. Imagine
being able to track someone's complex
language use in relation to disease state
over years, the effects of an intervention
on an autistic child's social functioning,
or predicting relationship outcome based
on how well people control both their
internal physiology (arousal, body temperature) and their expressions toward a
spouse during conflicts (e.g., [12]).
Algorithms are now able to handle
multiple sources of variability in a wide
range of collected data. Applications
such as speech recognition and computer vision demonstrate the capabilities of
handling this mapping from low-level
noisy signals to midlevel behavioral features, and, more frequently, researchers

large-scale multimodal data (Figure 1).
Signal processing is the keystone that
supports this mapping from data to representations of behaviors and mental
states. The pipeline first begins with
raw signals, such as from visual, auditory, and physiological sensors. Then,
we need to localize information coming
from corresponding behavioral channels, such as the face, body, and voice.
Next, the signals are denoised and modeled to extract meaningful information
like the words that are said and patterns
of how they are spoken. The coordination of channels can also be assessed
via time-series modeling techniques.
Moreover, since an individual's behavior is not isolated, but influenced by a
communicative partners' actions and
the environment (e.g., interview versus
casual discussion, home versus clinic),
temporal modeling must account for
these contextual effects. Finally, having
achieved a representation of behavior
derived from the signals, machine learning is used to make inferences on mental
states to support human or autonomous
decision making.

FIGURE 1. A schematic diagram of behavior generation and the mental state prediction process. First, a person's current mental state (Z ) affects their be-

havior (Y ). Raw signals are received (X ), on which signal processing is performed to produce representations of behavior (Yt ). Finally, machine learning
is used to predict a person's mental state (Zt ).
IEEE SIGNAL PROCESSING MAGAZINE

September 2017

189

Table of Contents for the Digital Edition of Signal Processing - September 2017