IEEE Computational Intelligence Magazine - May 2021 - 47

physical and physiological behavior, psychological states and
traits, and job performance. We develop a novel data mining
framework that can extract meaningful predictors from noisy
and incomplete data derived from wearable, mobile and social
media sensors to predict nineteen constructs based on twelve
standardized and well-validated tests. The framework can be
used to build a predictive model of outcomes of interest. We
validate the framework using data from 757 knowledge workers in organizations across the United States with varied work
roles. Our framework and resulting model provides the first
benchmark that combines these various instrument-derived
variables in a single framework to understand people's behavior.
The results show that our framework is reliable and capable of
predicting our chosen variables better than the baselines when
prediction includes the noisy and incomplete data.

comprised of a broad suite of sensing modalities for automated
modeling of individual physical, psychological, and job performance differences. Figure 1 shows a diagram of the specialized
sensors that were used to gather data in an unobtrusive manner.
The goal of these sensors was to collect information that was
representative of an individual's behavior, health, and mental
states. These sensors also collected data related to job and daily
activities (offline and online interactions, phone and computer
usage) both at home and at work. We enrolled a diverse cohort
of 757 knowledge workers performing varied roles at several
organizations across the United States.
Research Questions. We consider the following questions:
RQ1) How do we integrate data from different sensors and
modalities? RQ2) How do we develop machine learning methods that can deal with the challenges of varying levels of missingness and noise, and inter-individual, intra-individual, and

I. Introduction

earable devices are opening new avenues to
improve our understanding of our health and
well-being by tracking individual in-situ patterns
of activity and affect. These patterns can be
extended into precise and objective measures of health and
well-being, which can not only benefit an individual's health
goals, but also aid organizational efforts to promote wellbeing [1]-[3]. Detailed and continuous data collection can
yield valuable insights into health and well-being, e.g., stress,
sleep, work performance, and physical activity [4]-[7].
In recent years, literature that assesses well-being and its
effect on productivity has expanded. Well-being, personality,
and cognitive ability can affect job performance [8]-[15]. Personalized well-being assessment [1], [16] is also receiving more
attention due to the ubiquity of wearable devices.
Despite advances in computational methods that use wearable data to assess well-being, e.g., [1], [16]-[18], substantial
challenges still exist. Previous work has been limited by data
characteristics such as small samples, homogeneity (specific
demographics and work roles), or controlled environments
(within specific scenarios and locations). Additionally, predicting well-being, where productivity or performance is considered [19], becomes difficult due to scarcity of situational
(contextual) information [20], [21], privacy, or other concerns
[8]-[15]. Assessing both well-being and workplace performance requires machine learning strategies that overcome
these issues. In our work, we identify three main challenges: 1)
unobtrusive data collection using large samples of heterogeneous individuals in a wide range of work settings and geographic locations; 2) need for multi-modal sources for holistic
representations of physical and behavioral patterns; and 3)
missing and noisy values that stem from the nature of the data
and sensors and idiosyncrasies of individuals (e.g., gaps in measurements, variable compliance in wearing the devices, failure
of sensors) [22], [23].
In this paper, we report our approach in addressing these
challenges and building a machine learning pipeline for the
Tesserae Project [24]. In Tesserae, we implemented a system

Training

Test/Validation

Preprocessing

Heart Rate

Stress

Higher-Order
Network
PCA

PCA

Ensemble Components

Ensemble

Prediction

FIGURE 1 Model Diagram: We use a set of weak modality-based
learners to produce a strong prediction for each variable. Database
icons represent repositories of data that is fully anonymized and
ready to be processed.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021