Signal Processing - September 2017 - 191

computation with human computation.
Imagine a scenario in which an individual wants to observe the behavior of her
employees, patients, or students based on
video-recorded data. That person may
currently be limited to manually evaluating several random instances. Instead,
signal processing could be used to build
models for the behaviors of interest and
locate the instances in the data that are
most relevant, or salient, to an overall
judgment. One such recent approach is
to identify salient words using attentional
recurrent neural networks (RNNs) [17].

Measuring what we can't see or hear
Aside from overt signals that we freely
perceive with vision and audition, there
are covert signals that can reveal other direct insights into a person's mental state.
Consider the case of lying: when we lie,
generally our blood pressure rises, we
breath faster, our heart rate increases,
and we sweat (these are the signals that
a polygraph tracks), all while we attempt
to express behavioral signals of calmness
and honesty. This potential mismatch
between internal state and the related behavioral expression has clinical relevance
as well. Take, for example, the case of
an individual with moderate-to-severe
autism spectrum disorder who also displays self-injurious behavior. Such an
individual may not give outward indications of building agitation; but it has been
shown that, in certain cases, physiological
monitoring of arousal can indicate an impending episode, which would be critical
knowledge for care providers. While telemetric monitoring of physiological and
cognitive behavior has been considered
for years, we are (rapidly) improving the
sensing and computational tools to make
sense of these physiological signals and
act upon it in the moment. The great challenge currently is to develop robustness in
the sensors and signal processing.

Multimodal temporal modeling of
individuals and interactions conditioned
on context
Most psychological phenomena are complex and multifaceted, therefore requiring
a multimodal set of information for detecting and quantifying them. Integrating
multiple sources of data can afford us a

more complete view of the human state
but, at the same time, imposes challenges
in terms of information perception and
data fusion. Different modalities can be
complementary, redundant, or even conflicting. Multimodal patterns and interactions can also be situation dependent and
person specific. The conceptualization
of the acquired multimodal information
(e.g., through human-derived annotations, self-reports, metadata, etc.) can
help us identify the most relevant bits and
streams of information at a given timepoint and for a given situation.
To elaborate on these points, we will
draw an example from the family studies
domain, which investigates close relationships between family members and
romantic partners. Researchers in this
domain are particularly interested in the
way partners experience interpersonal
conflict since conflict affects quality of
life, mental health, and physical well-being.
Recent advances in mIoT and wearable
technology allow us to capture interpersonal conflict in everyday life via multimodal
sensing, enabling answers to a variety of
questions related to this phenomenon, i.e.,
when and why conflict occurs, whether it
is efficiently handled, and when it escalates.
Different people experience conflict
in different ways and through various
signal cues depending on their personality, family history, and previous
life experiences. For example, conflict
in avoidant people might be expressed
through changes in their physiology,
while in competing personalities such
episodes might be apparent through
acoustic and visual cues. This variability
is enhanced by other contextual factors,
such as the individuals' physical conditions (e.g., medication intake, caffeine/
drug/alcohol use) and locations (e.g.,
home, work, etc.). Taking these into account, reliable conflict detection systems
should consider a multimodal stream of
information (e.g., audio, language, physiology, etc.) complemented by contextual
data streams (e.g., global positioning system location, self-reports of nutrition and
substance use, measures of body temperature and acceleration); an example
of such ubiquitous sensing is [12]. Since
every individual and every couple have a
different baseline functionality, attention
IEEE SIGNAL PROCESSING MAGAZINE

September 2017

should be paid to establishing personalized models through knowledge-driven
priors (e.g., background of family aggression, previous relationships, etc.) and
data-dependent approaches (e.g., adapting system decisions based on data from
similar participants).
Beyond the typical multimodal feature
fusion, explicitly capturing the interaction of various modalities within person
and across people can be very informative. Aggregate statistics can only provide
information about the overall relationship between participants but not how it
evolved. This can still be useful; for example, previous work has demonstrated
that in diagnostic interactions for autism,
the psychologist-who is conducting a
semistructured interaction-must adapt
his or her behavior to the child with
whom they are interacting [15]. The
results showed that children with higher-severity autism spoke less, while the
psychologist spoke more; the child spoke
with higher prosodic variability, as did the
psychologist; and so on. In fact, the psychologist's prosodic features were at least
as predictive of the child's autism severity
as those same features from the child.
Forthcoming advances can help in understanding interaction through dynamic
modeling (the third BSP setting). It is
with such models that capture emotional
and behavioral evolution that computational tools will gain widespread validity
and utility. General mathematical models
that have been considered include dynamical systems equations, hidden Markov models, conditional random fields,
and long- and short-term deep neural networks. Knowledge-driven algorithms are
also being created. One such example is
the quantification of behavioral entrainment, or synchrony, which is the mutual
influence displayed between interacting
partners; this influence is said to be stronger for more positive interactions, but
human observers have great difficulty explaining "how" and "why" some pairs are
more entrained than others. Hence, Lee
et al. [18] have proposed a signal similarity metric that computes the convergence
of two interacting people with respect
to their representative signal-spaces,
i.e., principal component (PCA) space.
Generally, the vocal behavior of partners
191

Table of Contents for the Digital Edition of Signal Processing - September 2017