IEEE Computational Intelligence Magazine - May 2021 - 48

(office, home, and two portable), and social media
(Facebook). As part of our work, we identified
the most relevant subset of these features for each
of the nineteen variables.
While sensor data has been used to create
models of human behavior, including: daily
activities, mobility, [25]-[27], well-being [28],
and academic performance [17], to our knowledge there is no study that harnesses multimodal data sources into a unified comprehensive framework
that addresses data messiness and algorithmic challenges to
create a generalizable machine learning pipeline predicting
person-level behavior, physical and psychological wellbeing, and job performance.
To create a generalizable joint model of the nineteen constructs, we implement several strategies. First, we consider
various imputation approaches as well as co-dependencies
among the variables for both feature selection and prediction
(RQ1). Unlike other approaches that deal with missing data
for longitudinal scenarios (e.g., [22], [23]), our approach does
not require complex likelihood-based techniques. Second, we
use a fusion technique to synthesize multi-modal sensorderived features (RQ1). Third, we consider an ensemble
learning technique to incorporate various machine learning
models (RQ2). Fourth, in addition to our data policy and
machine learning design, we use higher order networks
(HON) [41] to obtain descriptions of each individual (RQ3).
Finally, to ensure generality we perform 5-fold cross-validations at all stages of the model creation: feature selection,
dimensionality reduction, hyper-parameter tuning, training,
prediction, and tests.
In summary, our contributions are as follows: 1) We create a
framework whereby noisy, heterogeneous, multi-modal data
can be fused without the need for highly specialized curation;
2) We provide a benchmark that leverages data fused from the
modalities to produce more integrated predictions of human
behavior than existing techniques; and 3) We verified experimentally the predictive capability of our approach using data
from our longitudinal study. The results show that our model
perform favorably with respect to theory-driven baselines. We
verify the results using various reliability tests.

In order to acquire a comprehensive view of an
individual's physical attributes, psychological
characteristics, personality, and job performance,
we used a battery of psychometric surveys to obtain
the target variables.
inter-sensor variance? RQ3) Are individual job performance,
well-being, and personality predictable from these sensors?
Critically, real-world wearable data can also be affected by
(ir)regularities, temporal variations, and differences in an
individual in addition to missing values, noise and other
issues mentioned before. Thus, generalizability issues arise in
models trained on samples from this data because the training sample distribution may be different than the test sample
distribution used in the final applications. While sensor data
has been used to predict and assess human behavior and
well-being [25]-[28], these predictions are done on highly
curated and often homogeneous data. Such models may be
overly-optimistic about what can be achieved in real-world,
messier scenarios.
We address these data issues and create a machine learning
solution to our joint prediction problem. Specifically, we model
nineteen survey-based variables [29]-[40]. The 927 predictors
(features) were obtained from sensors assigned to each participant in Tesserae: a wearable (Garmin Vivosmart 3), a phone
agent (an app for iPhone and Android), four Bluetooth beacons

TABLE I List of dependent variables.
TYPE

SUBTYPE

VARIABLE

Job Performance

Task

IRB [29]
ITP [30]

Org. Cit. Behavior

OCB [31]

Deviance [32]

Interpersonal
Organizational

Psychological

Cognitive [33]

Vocabulary
Abstraction

Personality [34]

Extraversion
Agreeableness
Conscientiousness
Neuroticism
Openness

Affect [35]

Positive
Negative

Health

Trait Anxiety [36]

Anxiety

Consumption

Alcohol [37]

Activity

In order to acquire a comprehensive view of an individual's
physical attributes, psychological properties, personality, and job
performance, we used the battery of psychometric surveys to
construct our ground truth variables listed in Table I. Surveys
were administered at the beginning of the study, and shorter
versions periodically over the first 60 days of this year-long
study. We present the analysis for the initial battery of surveys
(results for the shorter versions are presented in [42]).

Tobacco [38]

A. Job Performance

Physical [39]

We considered five variables that assess job performance from
three perspectives: task performance, organizational citizenship

Sleep [40]

48

II. Background

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
