IEEE Computational Intelligence Magazine - May 2021 - 55

Given the discretized heart rate data, the
output is the conditional probabilities of
each individual
P (x t|x t - n, f, x t - 1) =

I(x t -n, f, x t -1, x t)
I(x t -n, f, x t -1)

Our experiments evaluated the stability of the
predictions. An external team validated our model
on a sub-cohort of participants whose data was
withheld from our team during model development.

where n denotes the network order, I(x) indicates the number
of occurrences of x.
HON applies a low-pass filter to ternary relations
among the selected patterns derived from the discretization
step in Figure 2. For instance, consider the heart rate timeseries illustrated in Figure 3. An algorithm that only identifies first order relations could describe the probabilities of
going from a heart rate of 90 bpm to 100 or to 120. HONs,
which can identify more than first order relations, can
describe different probabilities for heart rate transitions
that go from 80 to 90 to 100 (or to 120) compared to heart
rate transitions that go from only 100 to 90 to 100 (or to
120 bpm). PCA was used to reduce feature dimensions to a
target n_component = 5 from 727 original features (transition
probabilities). HON captures transition probabilities across
individuals, while heart-rate and heart-rate variability are
within subject features. Thus, we capture both general and
individual heart patterns.
We investigated different small orders (1-5) of HONs and
chose order 2. The number of transition probabilities exponentially increases as the order of the network increases, which lead
to a sparsity problem. In particular, most transition probabilities
of each individual might be zero as the order increases. As the
number of elements in a possible transition increase, the transition may not be associated to a participant.
E. Imputation

Two approaches were used: (1) a theoretically-driven approach
that attempted to fuse data across multiple sensing streams using
the knowledge of subject-matter experts (e.g., sleep can be
fused between the wearable, and smartphone [139]) and (2) a
data-driven approach that can vary across the various features
(impute via the mean, impute via zeros, etc.). For the joint
-prediction of the physical, psychological, and job-performance
variables, we also performed sensor-wide imputation. For this
purpose, we considered the data from one stream and performed clustering on it. This allowed us to impute missing data
in one stream from data in another based on the relationships
between sensor streams. Other techniques applied include mean
and median value imputation. We also performed data imputation using individual rolling means, i.e., individual mean value
up to the specific moment. If there was no record at all, we used
the global mean.
The level of sparsity was a critical challenge for the
phone agent data at the raw data level. However, this was
overcome by carefully selecting regularity-based features.
Regularity features can capture rhythms and routines within
a participant, namely the patterns within hourly phone

usage, physical activity and mobility across the participant's
time series. Additionally, we had to deal with sparsity for
heart rate variability (HRV) when the size of the window
used to compute the HRV was not adequate over the
5-minute windows [140]. Some sparsity was also due to data
quality issues. Since HRV windows are calculated using
Beat-to-Beat-Interval (BBI), many windows did not have a
minimal number of BBI readings. This was due to inconsistencies in the data updates from the wearable. HON selection also had sparsity constraints, as higher order networks
provided no further information than lower order ones.
Namely, we combined the features from each stream/data
source and then we applied our regression models for prediction purposes.
F. Fusion

For the joint prediction of the physical, psychological, and jobperformance variables, we also used a feature fusion method to
combine the various modalities. The features from each stream/
data source are combined and fed into our regression and classification candidates for the automated selection of the model
at the model selection step. Thus, we capture moments from
the distribution of features that provide a summary of each of
the modalities. For numerical features, we use summary statistics: mean, median, standard deviation, minimum, and maximum of the distributions. For time series data, we use the
features extracted from HON and from other summary statistics. For the PhoneAgent we considered regularity based highlevel representations, as well as the imputed values that help
model building at the component-of-the-ensemble level. The
specific prediction models as well as the relevant features were
selected by the cross-validation process. For the final ensemble,
we considered a model selection.

HR 80

HR 100

HR 90 HR 80

HR 100

HR 120
HR 90 HR 100

FIGURE 3 HON-Heart rate case example. The node HR90 is broken
down with HONs by including information about the path. Thus,
HR90 originated from HR80 will have different probabilities to link to
HR110 (or HR120) than HR90 originated from HR100. The arrows
have different width to represent the relative difference in probability.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021