IEEE Computational Intelligence Magazine - May 2021 - 56

G. Model Selection

Using the elements described so far, we built the components of the ensemble learning model by combining the
HON features (heart and stress), heart rate, social media, beacons, phone agent, and wearable. We did so with the following steps:
1)	Feature pre-selection. We use both the sequential exploration of various combinations of features to identify a set

Algorithm 1 Joint model.
Input: Multimodal Data D
Output: Predictions
   1: Divide D in training and validation sets T, V
   2: Use T to apply feature selection (top 20 features per modality
with highest correlation to the nineteen constructs) to select
candidate features
  3: for each ground truth variable do
   4:   for each parameter-set do
  5:   for fold =1 to 5 of T do
  6:    for each candidate-component do
   7:     Predict on current fold using candidate-component
trained on the remaining folds
  8:    end for
  9:  end for
 10:  
SelectedComponent ! candidate with highest score across
the folds
 11:  Add SelectedComponent to Ensemble
 12:  Add the fold-wise SelectedComponent predictions F
  13:   end for
 14: end for
 15: model ! train the ensemble on T
  16: Set predictions P ! Predict(V,model)
 17: return F, P

TABLE V Performance SMAPE (%)-sensors vs. baseline.

56

of predictive features per construct and the social media
anonymization of features described before.
2)	Relevance-based feature selection. Each specific technique
uses an a priori relevance (measured by correlation) on
the training set (linear or non-linear correlation).
3)	Model selection. Automated machine learning methods are
applied to decide the best set of features along with the
best classifier/regressor per construct.
4)	Proxy ground truth. We considered the predicted values for
AUDIT and OCB, due to higher predictability, in order
to perform prediction of other values. We then use these
predictions and loop back to the previous step.
Dimensionality reduction through principal component
analysis was applied on HON construction (both stress and
heart rate) and on social media data. The candidate-components were described in Section V-A, where a component is
selected as a member of the final predictor ensemble. The main
training and predictions are shown in Algorithm 1.
The data and code are in the process of being released
through the supervision of IARPA. In order to protect human
subjects, only non-identifiable information will be released. We
will make the data available through the Open Science Foundation and a corresponding Data Use Agreement (DUA).
Information can be found at https://tesserae.nd.edu.
VI. Experiments

We evaluate our approach using four sets of experiments. First,
we investigate the performance of our model when compared
to a baseline constructed with estimators derived from the
ground truth values as detailed below. Second, we verify the
bivariate and discriminant criterion validity. Third, we verify
the model reliability under 5-fold cross-validation to ensure
generalizability. Finally, an external team validated our model
on a sub-cohort of participants whose data was withheld from
our team during model development.

VARIABLE

SENSOR-BASED

BASELINE

IRB

3.8

7.9

ITP

4.6

9.4

OCB

6.8

14.2

A. Data Setup

Interpersonal Deviance

18.7

32.9

Organizational Deviance

14.8

28.5

Abstraction

6.4

13.4

Vocabulary

4.2

8.8

Extraversion

8.3

17.5

Agreeableness

5.7

11.6

Conscientiousness

6.8

14.2

Neuroticism

12.6

26.0

Openness

6.4

13.2

Positive Affect

6.6

13.5

Negative Affect

11.4

22.2

Anxiety

10.1

19.9

We consider the twelve standardized tests administered in our
initial ground-truth battery that contained all 19 dependent
variables used for prediction. The independent variables came
from various data sources (a wearable, a phone app, four beacons, and social media data) as described in Section IV-A.
1)	Data Selection and Feature Set: We perform preprocessing of
all the streams previous to fusion of the features as
described in Section V-B.
2)	Metrics: We use the Kendall's x correlation coefficient
which is a non-parametric measure of correlation based on
rank statistics and, thus, assumes no specific structure of the
data. Specifically, to compute x scores, we apply the General Monotone Model (GeMM) [141].

Alcohol

30.6

70.4

B. Results

Tobacco

92.2

195.6

Physical Activity

30.8

68.9

Sleep

13.4

27.3

We test our framework using 4 sets of evaluations:
1)	Validation vs. Theory-Driven Baseline: In this set of experiments, we verify that the performance of our model is

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021
https://tesserae.nd.edu
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
