Signal Processing - September 2017 - 193

Behavior tracking and intervention
Today, metabolic health monitors such
as Fitbits are very popular; these monitors are based on extremely simple signals and provide limited metrics, e.g.,
inferring full body movement and thus
exercise from only the motion of the
arm. But many people use them as a
reminder to better their own health via
behavioral regulation. Thus, they are a
great example of how one might scale
a response to meet the demands of the
global community.
Once we quantify behavior objectively using signal processing and machine
learning, we can monitor those behavioral
constructs over time. This opens up immense opportunities to not only assess
the trajectories leading up to and after
diagnosis but also to assess the response
to a pharmacological or behavioral intervention. Furthermore, we can in turn utilize these relevant behavioral measures to
create novel interventions. For example,
once we adequately characterize "atypical" speech prosody as it relates to developmental disorders, we can create
personalized computerized interventions
that would otherwise not be possible.
One can imagine a scenario in which
behavioral monitoring of self and others
is incorporated into ubiquitous computing devices that provide feedback to the
wearer; researchers are already considering the use of tech-glasses to assess the
emotion of the person they are interacting
with, but much more will be possible as
this field evolves.
One example utility of monitoring
behavior is recent work aiming to monitor a patient's mental health over time
based on phone call recordings [20]. In
this paradigm, a person who was seeing
a psychiatrist for symptoms of a mental
health disorder (i.e., major depressive disorder, bipolar disorder, schizophrenia, or
schizoaffective disorder) would frequently call into an automated system and leave
an update on their status. The patients responded to automated questions on how
they were overall, what was going well,
and what had been bad. The interdisciplinary team employed vocal analytic and
natural language processing techniques to
build global and person-specific models
of how a person's speech characteristics

related the psychiatrist's opinion of their
emotional state. Initial findings suggest
that speech features are able to quantify,
to an extent, how well someone is doing. Relatedly, social network analysis is
emerging as a potential means to track a
person's state over time at scale.

Providing feedback to
the health-care professional
Another avenue to impact mental and
behavioral health is through providing
objective feedback to the provider about
their own actions, serving as an enabler
of training. Providing clinicians an indepth overview of therapy and diagnostic sessions empowers them to both
track patient progress as well as alter
their own approaches.
A good candidate for such a system is
psychotherapy interactions, due to their
overall importance in treating mental
health as well as their semistructured nature. These interactions typically occur
between a limited number of participants
(i.e., a therapist-patient dyad) where the
primary mode of communication is verbal. The words that are spoken can be analyzed for important counselor behaviors
such as reflections (restating what the
client says to demonstrate understanding) or the patient's commitments to behavioral change, both of which indicate
positive addiction counseling sessions
[21]. Beyond the words that are said, the
tone of voice, emotion, and politeness
can be critical feedback for a counselor.
To enable this goal, an automatic system has been developed that segments
speaker utterances, assigns them based
on automatically determined role, performs transcription, and, finally, predicts
behavioral codes. Developing such a tool
for clinical use requires incorporating
user interface and experience design elements that make the experience both useful and intuitive; a tool presented in [22]
allows clinicians and supervisors to select individual utterances and review the
automatically transcribed words as well
as the behaviors predicted to be present
in that particular utterance. It also displays session-level gestalt behaviors and
behavioral counts to provide a high-level
overview of important measures such as
therapist empathy.
IEEE SIGNAL PROCESSING MAGAZINE

September 2017

Challenges and opportunities in
formulation and implementation
While we have established that behavioral modeling from signals is a viable
path forward for mental health, as the
field grows there are critical questions
that must be answered. Some of these
challenges are unique to each problem,
while others are broadly applicable to
this interdisciplinary research.

Data collection and modeling
All stages of the behavioral signal processing pipeline are intertwined; even the
problem motivation is not independent
of the technical ability and process for
achieving a goal. Therefore, it is also imperative that all aspects of BSP problem
design are adequately addressed. The
first stage is data collection, for which
we desire data of high quality and high
consistency. Since behavior is inherently
multimodal, we must collect multimodal
data. This collection should be ecologically valid or natural and not affecting
the behavior itself. Another critical factor
is time. One of the great contributions of
technology will be longitudinal, time-continuous monitoring of behavior. Consider
that a psychologist typically has one hour
to interact with a child as part of an autism
diagnostic evaluation. That child may simply be having a bad day, so assessing progress at the individual level is obstructed.
Alternatively, for privacy or data storage
reasons, sampling often needs to be done
in a time-sensitive (i.e., "sparse") fashion.
The second and third stages are analysis (deriving behavioral features) and
modeling (mapping features to behavioral constructs), for which the core challenges stem from the extreme complexity
of human behavior and the uncertainty
in the measured signals. No two expressions will produce identical signals due
to variability within a person and across
individuals, as well as external sources
of noise. For example, we never actually speak a word the same way twice:
intonation will vary with mood, certain
syllables will be spoken slightly faster or
slower, and the channel conditions can
change between recordings. Computational systems must be robust to such
heterogeneity and further must translate across data sets; for example, it has
193

Table of Contents for the Digital Edition of Signal Processing - September 2017