IEEE Computational Intelligence Magazine - May 2021 - 58

and job performance constructs. The most challenging variables in terms of reliability are anxiety,
tobacco consumption, interpersonal deviance, and
agreeableness. The lower limit for the performance
ranges can be explained from the difficulty of
modeling social constructs in general and human
performance in particular [19]. These reliability results were
verified externally.
Our work provides a realistic assessment of the performance
of prediction algorithms. This was done by respecting the
nature of the data. We did not curate nor select data objects for
optimal performance. Instead, we worked with the full original
dataset which included individuals with both full and partial
sets of features and modalities. While we experimented with
neural networks techniques as candidate components (not
included in this article to simplify exposition), the models with
highest weight in the final ensemble were always parsimonious
models. This is consistent with previous observations [144] that
simple models perform as well as more complex models, in an
empirical realization of the Occam Razor in the social sciences.
The first week or two add noise (low/irregular compliance) yet
our model's performance is stable despite the missingness. Our
model provides a simpler alternative for dealing with missingness without the use of complex likelihood-based or similar
techniques (e.g., [22], [23]).
Our work focuses on individual traits assessed through psychological constructs. Specific context about emotionality may
be an area for future work. Sentiment analysis [107], [145],
[146] could be applied for feature extraction. Another area of
possible work is adding features detailing context, e.g., by combining text features with commonsense knowledge.

Realistic models on messy data may be preferable
to overly optimistic models. Our work provides a
benchmark on this type of predictions.
distribution by running a 5-fold cross-validation and thus,
all the values obtained are created from independently built
models. Our assessment goes beyond single-dimensional
summaries of performance to include measures of variation,
confidence, and distributional information. As we can see in
the figure, the variables for which our framework performs
the best are physical variables, followed by job performance
and psychological constructs. Tobacco use and job performance are the most challenging constructs to predict.
5) External Validation-Totally Unknown Cohort: We provided an
external evaluation team with a pipeline and data to corroborate our results on a sub-cohort of participants whose
data was totally unavailable to us during model development. This validation was administered by the Testing and
Evaluation Team of the Project Sponsor. The independent
evaluation was performed on variables other than job performance and can be seen in Table VI. These results are
-consistent with what we report in Figure 4. The evaluation
shows x scores in similar ranges as those obtained in our
experiments Figure 4. It also shows that variables such as
agreeableness, neuroticism, openness, affect (negative affect
in particular), and tobacco consumption are the hardest
variables to predict, and some variables have less stability
than others. However, despite this challenge, some of these
variables (e.g., openness, positive affect), have a mean x performance greater than 0.14. A few variables show better
performance in our 5-fold validation (Figure 4) and a few
show better performance in the external evaluation. When
assessing these results the most conservative values should be
considered as reference.
VII. Discussion

The experiments suggest our model is stable with non-trivial predictive performance that is better than construct-based alternative
baselines. The performance of our technique is usually better for
physical variables of well-being such as alcohol consumption,
sleep, and physical activity. The performance is also competitive
for psychological and job performance variables. We verified the
significance of our sensor-based predictions compared to a participant-oriented baseline to predict job performance variables. The
linear-mixed model predictions based on our estimates produced
better x- scores than predictions based on survey estimates. Thus,
our framework has better bivariate criterion validity.
The discriminant validity analysis show that our framework
sufficiently identifies the various constructs with small absolute
values [143] for the correlations 6- 0.21, 0.2@ . The reliability
analysis shows that the model is reliable as a reflection of the prediction performance. Roughly speaking, physical variables are
the most reliably sensed and estimated, followed by psychological

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021

VIII. Conclusions

Assessing workplace performance, psychological, and physical
characteristics of individuals usually relies on existing full traditional questionnaires, on subjective evaluations. Furthermore, current predictive techniques are effective in limited
situations including small subsets of variables, subsets of highly
curated data, or with a focus on a few variables without a
global overview of an individual. In this paper, we presented
the first modeling framework and benchmark that leverages
sensor data from multimodal sources to jointly predict psychological, physical and physiological, and job-performance
constructs. We used traditional social and psychological questionnaires to create the ground truth variables. We used
objective mobile and personal sensing data from social media,
phones, wearable and beacons as predictors and offer new
insights into behavioral patterns that distinguish our various
constructs. We presented results from our study of 757 information workers collected over a period ranging from 15 days
to 60 days. We created a global ensemble learning algorithm
that takes advantage of various data mining techniques and
feature extraction approaches to achieve our joint prediction
problem on messy data. Our results indicated that our modeling framework allows for a prediction performance above

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021