IEEE Computational Intelligence Magazine - May 2021 - 52

beacons (home, office). This provided details of interactions
using the strength of the signal as described next.
Beacons: Gimbals. Beacons are low energy devices that
transmit and receive Bluetooth signals to and from other
devices. We used four Gimbal beacons per participant in our
study. Two beacons were the static Gimbal Series 21, with
one placed at the participant's home and the other placed at
the participant's workplace. The other two Series 10 beacons
are small, coin-size, mobile beacons that participants carried,
one in their key-chain or wallet, and one in their backpacks
or purses. Beacon signals were detected by the phone
through our PhoneAgent app which uses the Gimbal API
library to detect proximity. When a PhoneAgent enabled
smartphone approaches a beacon, the phone will detect a
Bluetooth signal and will record the signal strength which is
inversely proportional to the distance between the phone
and the beacon. The beacons provided information about the
location of an individual relative to their home, work, or to
other participants. This allowed us to derive features that
describe the mobility of individuals and other daily locationbased routines while hiding actual physical locations. These
features were stored by the PhoneAgent into a local server
and on Gimbal servers.
Social Media. During the recruiting process, we requested
read access to the participant's accounts on Facebook and
LinkedIn. As with all of the other data sources, we anonymized
their data but, in the case of social media, we also modified the
data so as to avoid storing raw information that may affect privacy. After data collection from social media, we applied feature
extraction techniques and stored only the anonymized features.
For the present analysis, we considered 5,075 raw features computed from participants Facebook data; most of these were
n-grams (words/phases) of posts. However, a feature selection
step was applied to select only the relevant features, as detailed
in Section V-C. These raw features corresponded to a variety of

TABLE IV Low-level sensor-derived features.
SOURCE

SUB-MODALITY

Wearable

Higher Order Network-Heart Rate

Higher Order Network-Stress

Heart Rate

Phone App

Beacon

Social Media

Other Physical

Physical Activity

Context

User State

Phone Usage

Regularity

580

Work Activities

Other

Home Activities

5
200*

*=POST PCA.

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021

categories-1) psycholinguistic attributes [138] (that captured
language usage across keywords related to affect, cognitive attributes, perception, interpersonal focus, temporal references, biological concerns, and social and personal concerns), 2) open
vocabulary n-grams (the 5,000 most frequent uni-, bi-, and trigrams used by the participants), 3) sentiment in posts, and
4) social capital (e.g., by measuring check-ins to places, posting/
sharing updates, uploading media, changing relationship status,
and hanging out with friends).
B. Predictors

We used a total of 927 candidate features (filtered out later
with dimensionality reduction and feature selection techniques,
as detailed below) based on the sensor data from the PhoneAgent, Garmin wearable, Gimbal beacons, and social media.
We extracted additional information from two wearable generated time-series per participant: heart rate and stress measurements. We used these time series as separate components to
extract features that facilitate discriminatory prediction based
on signatures extracted using a higher order network (HON)
approach with one HON per time series. We also used the
heart rate to build an additional component for the ensemble
using a special representation for the time series. Table IV
details the number of features used per data source. For features
collected as time series we computed the daily mean, median,
mode, minimum, and maximum.
Examples of features collected from Garmin (through the
Connect API) include sleep staging (duration in light, deep, and
REM sleep) and bed time, daily step counts, daily floors
climbed, physical activity (duration of light, medium, heavy
activity), calories burned, and stress level (in range 0-100).
Examples of features collected by the PhoneAgent include
phone usage (number of screen locks and unlocks, duration of
locks and unlocks, etc.) and daily aggregations of physical activity such as mobility features (e.g., places visited, distance traveled, duration of sedentary state, driving and biking time). The
PhoneAgent also collected fine-grained data from the wearable,
e.g., heart rate, sleep, stress and steps. We computed time series
features at a daily level (which we call epoch-0) but also in
epochs within the day: early morning (12AM-9AM), day
(9AM-6PM) and evening (6PM-12AM). We used the epochs to
identify differences of behavior for the times that are associated
with sleep, work, and nightly activities.
Examples of features collected through the beacons
include various measurements of closeness of the static and
mobile beacons. These features in their raw form do not provide direct insights about the participants' activities, but in
combination with the type of beacon and the duration of the
interactions we can capture information such as the time
spent at work (total duration a participant spends at work
from the first to the last sighting of the work-beacon), the
time spent at desk (percentage of the time a participant
spends at their desk), and the number of breaks taken away
from the desk that exceed 5, 15 and 30 minutes (captured by
gaps in beacon sightings).

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021