IEEE Computational Intelligence Magazine - August 2020 - 55

analysis), and acoustics (intonations and prosody). During data
acquisition, videos were analyzed by automatic face detection
to verify whether a unique speaker is present. Moreover, only
the videos where the speaker's attention is exclusively
towards the camera were kept. The number of videos acquired from each channel was restricted to 10 to avoid bias
and all videos must have correct transcriptions provided by
the speaker. The quality inspection has been made by 14
expert judges, and 3,228 videos were selected from the 5,000
initially gathered.
The 3,228 videos were then segmented into 23,453 annotated pieces, where each segment contains a manual transcription aligned with audio to phoneme level. The annotation of
CMU-MOSEI closely follows the annotation rules of the
CMU-MOSI [39] dataset. In particular, sentences were annotated for Ekman's six emotions, that is happiness, sadness, anger,
fear, disgust and surprise, on a [0,3] Likert scale for the presence
of emotion. As such, 0 stands for no evidence of x, 1 for weakly
x, 2 for x, and 3 for highly x. With respect to sentiment evaluation, a [-3,3] Likert scale was used such that: -3 is highly negative, -2 is negative, -1 is weakly negative, 0 is neutral, 1 is
weakly positive, 2 is positive, and 3 is highly positive. Note that
in this paper, we do not use the annotation for sentiment evidence. As stated in [17], the annotation was carried out by 3
crowd-sourced judges from Amazon Mechanical Turk platform,
where judges were provided with a 5 minutes training video
on how to use the annotation system in order to avoid extreme
annotation, and all judges were master workers with an approval rate higher than 98%.
Note that as in CMU-MOSEI each of the 3,228 video
transcripts contains an average of 7.3 utterances, and in DAICWOZ, the 138 interview transcripts contain an average of 90
utterances, we randomly selected 517 transcripts from CMUMOSEI to reduce imbalance between datasets.
4.3. Learning Setups

With respect to multi-task learning, the task-specific LSTM
layers are trained alternatively using the entire training split. As
an example, consider the training of the shared-private multitask network for depression level regression and emotion
intensity regression: SP MT. DLR+EIR. The DLR-specific
LSTM layer, the shared-LSTM layer, and the corresponding
attention fusion network and fully-connected network are
trained for N DLR epochs without updating the weights of the
EIR-specific layers. For the next N EIR epochs, the EIR-specific
LSTM layer, the shared-LSTM layer, and the corresponding
attention fusion network and fully-connected network are
trained without updating the weights of the DLR-specific layers. Here, N DLR and N EIR are treated as hyperparameters. We
go on training the network in this alternating fashion till a
maximum number of iterations N total (the total number of
times the shared-LSTM layer is trained) is reached. The model
that shows best performance on the development split over all
iterations is chosen for testing. The pseudo-code for our training procedure is shown in algorithm 1.

Algorithm 1 FS/SP/ASP MT. T1 + T2 training.

1: n total ! 1
2: while n total 1 N total do
3:
for n T1 ! 1 to N T1 do
4:
Update T1 -specific and Shared weights
n total ! n total + 1
5:
6:
for n T2 ! 1 to N T2 do
7:
Update T2 -specific and Shared weights
n total ! n total + 1
8:

The architectures have been implemented with Keras11 and
hyperparameters have been optimized through grid search.
Note that all learning models are trained on the basis of stratified 5-cross validation, thus keeping the data distribution
between training, development and test datasets. In particular,
the best of the 5 models over the development set is applied to
classify/regress the examples in the test set.
5. Results

In order to test our hypothesis, we perform a series of experiments for three different tasks: (1) Depression Level Regression
(DLR), which aims to assign a value between 0 to 24 (that is,
the PHQ-8 score) to a given patient interview transcript, (2)
Depression Level Classification (DLC), whose objective is to
identify the correct discrete class of depression level (Noneminimal, Mild, Moderate, Moderately severe, Severe), and (3)
Emotion Intensity Regression (EIR), which regresses a [0-3]
value for each of the six Ekman emotions (happiness, sadness,
anger, fear, disgust and surprise) for a given user transcript.
Five different models serve as baselines. That is, each task is
first modeled as a single-task problem, and two unaware-emotion multi-task (fully-shared and shared-private) architectures
are implemented that combine both DLR and DLC.12 Three
different combinations of emotion-aware multi-task frameworks are tested, for each one of the three theoretical models
(fully-shared, shared-private and adversarial shared-private): (1)
DLC combined with EIR, (2) DLR combined with EIR, and
(3) DLC combined with both DLR and EIR.13
To evaluate regression/classification results, we use wellknown evaluation metrics that are standard for depression level
estimation [40]: (1) Accuracy, F1 score and Matthews Correlation Coefficient (MCC) for classification; (2) Root Mean
Square Error (RMSE), Mean Average Error (MAE), Coefficient of Determination (R2) and Symmetric Mean Absolute
Percentage Error (SMAPE) for regression. In particular, we
include two other metrics (Over and Under), that complement
Accuracy and evaluate how much a learning model over-evaluates (Over) or under-evaluates (Under) the correct result.
Such metrics are important to understand the behavior of
learning models. But, as far as we know, they are not presented in related works. For classification, Accuracy, Over and
Under sum to 100% and are defined in equations 2 and 3. For
11

https://keras.io.
These baselines correspond to the 5 first rows of Table 2.
These models correspond to the 9 last rows of Table 2.

12
13

AUGUST 2020 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

https://keras.io/

IEEE Computational Intelligence Magazine - August 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2020