Signal Processing - September 2017 - 190

are extending the mapping to higher
level psychological constructs like affect and empathy. Recent technological advances in signal processing and
machine learning have enabled new
opportunities, while other possibilities
will require continued advancement in
fundamental mathematical foundations
and algorithm development.

Scientific opportunities
Three settings for BSP
In moving from qualitative descriptions
to quantitative representations, there
are three primary settings in which signal processing and machine learning
are used to measure human behavior
(as shown in Table 2). The first takes a
first-person view of inferring what is happening in one's mind in relation to behavioral (unobtrusively observable) and
bodily (measurements often requiring at
least some level of intrusion) signals: the
mind-body connection. The current standard is to use self-report or human observation of specific behaviors, and then
condition on known experimental variables-for example, measuring behavior/
physiology during lying and truth-telling
in a scientific study of deception. Mathematical formalisms can also determine
the possibilities and limitations contained
within signals such as the observability,
controllability, and stability of an underlying state description.
A closely related experimental setting
uses quantified behavior to understand
the process by which humans perceive
and make decisions. Signal processing can provide measures that scale, are
quantitative and objective, and are consistent across time. Moreover, if we can
identify the behavioral building blocks
for perception, then we can begin to ask
how instances are aggregated and weighted

toward a global decision: is a single salient
behavior most critical to a perceptual construct, or are all relevant events summed to
arrive at a decision?
The third setting is that of interaction between multiple humans or
humans and machines. Behavioral scientists are interested in systematically
modeling the dynamics of multiparticipant processes but often struggle
to fully capture the inherent complexities. A rigorous signals and systems approach to human interactions is needed
to capture the complexities of interaction dynamics as well as the inherent
uncertainty in all facets of the process
(perception/cognition/action) and ultimately to build the mathematical
foundations for combining human and
machine computation such that machines augment human capabilities, not
simply replace them.

Interpretable representations
One of the primary reasons that signal
processing is critical to behavioral computation is interpretability. It is often not
enough to simply make a decision in
health care, but the end-to-end system
must explain how it arrived at such a decision. Decisions on a person's health are
so vital that even if a system is able to diagnose a disorder with 100% accuracy, if
that system is a black box such that the
decision-making processing is unclear,
health-care providers are highly reluctant
to trust it. And rightfully so, since there
are certainly pitfalls to which machinelearning approaches can succumb.
One way to address this is through a
top-down approach, which incorporates
human knowledge into feature generation and modeling by taking into account
the structure of the data as well as the
phenomenon that is being modeled. For
instance, consider the case of measuring

190

Description
Studying produced behavior in relation to known internal or
contextual variables.

Human perception

Relating the produced behavior to human perception to explain
perceptual processes.

Interaction

Quantifying the give-and-take behavioral dynamics that occur
in human-human or human-machine interaction.

IEEE SIGNAL PROCESSING MAGAZINE

September 2017

Aiding human decision making
Much of the research community's focus
has been on developing the core methods
of behavioral sensing; however, computational methods don't need to replace
humans but instead can augment their
capabilities especially in complex clinical
decision making and in understanding
its impact. A great challenge going forward continues to be interfacing machine

Table 2. Three settings for BSP.
Setting
Mind-body connection

vocal arousal, or excitement, which is
generally expressed vocally by a higher
pitch, intensity, and high-frequency energy content. This reliable observation
has been incorporated into a rule-based
algorithm that has matched state-of-theart performance in cross-corpus vocal
arousal recognition [13]. Similarly, we
can quantify affect in any text through
semantic similarity metrics by leveraging
a small set of seed words that have been
carefully annotated by human experts
[14]. Another knowledge-inspired approach that will be discussed in later sections is that of quantifying what is atypical
about the speech of individuals with autism [15]. Top-down generative approaches dominate the neurosciences because
they afford a mechanistic understanding
in terms of low-dimensional biophysical
constructs, and thus signal processing
techniques have potential for strong contributions (for example, in modeling of
clinical trajectories in the emerging field
of computational psychiatry [16]).
Despite their advantages, top-down
approaches might not be able to take into
account the entire variability of the data
space, which is more likely to be incorporated through data-driven representations. Toward this direction, unsupervised
clustering and feature-learning techniques
as well as deep and recurrent neural network approaches have been proposed.
However, risks with bottom-up approaches
include the limited presence of human experts guiding these engineering efforts and
the risk of overfitting to inadequate data-
either in terms of quantity or quality (e.g.,
variable and/or bad recording conditions).
Semisupervised learning might be an intermediate solution that combines labeled
and unlabeled data to build better learners
and, at the same time, incorporates directly
the knowledge if human experts.

Table of Contents for the Digital Edition of Signal Processing - September 2017