IEEE Computational Intelligence Magazine - August 2020 - 55
analysis), and acoustics (intonations and prosody). During data
acquisition, videos were analyzed by automatic face detection
to verify whether a unique speaker is present. Moreover, only
the videos where the speaker's attention is exclusively
towards the camera were kept. The number of videos acquired from each channel was restricted to 10 to avoid bias
and all videos must have correct transcriptions provided by
the speaker. The quality inspection has been made by 14
expert judges, and 3,228 videos were selected from the 5,000
initially gathered.
The 3,228 videos were then segmented into 23,453 annotated pieces, where each segment contains a manual transcription aligned with audio to phoneme level. The annotation of
CMU-MOSEI closely follows the annotation rules of the
CMU-MOSI [39] dataset. In particular, sentences were annotated for Ekman's six emotions, that is happiness, sadness, anger,
fear, disgust and surprise, on a [0,3] Likert scale for the presence
of emotion. As such, 0 stands for no evidence of x, 1 for weakly
x, 2 for x, and 3 for highly x. With respect to sentiment evaluation, a [-3,3] Likert scale was used such that: -3 is highly negative, -2 is negative, -1 is weakly negative, 0 is neutral, 1 is
weakly positive, 2 is positive, and 3 is highly positive. Note that
in this paper, we do not use the annotation for sentiment evidence. As stated in [17], the annotation was carried out by 3
crowd-sourced judges from Amazon Mechanical Turk platform,
where judges were provided with a 5 minutes training video
on how to use the annotation system in order to avoid extreme
annotation, and all judges were master workers with an approval rate higher than 98%.
Note that as in CMU-MOSEI each of the 3,228 video
transcripts contains an average of 7.3 utterances, and in DAICWOZ, the 138 interview transcripts contain an average of 90
utterances, we randomly selected 517 transcripts from CMUMOSEI to reduce imbalance between datasets.
4.3. Learning Setups
With respect to multi-task learning, the task-specific LSTM
layers are trained alternatively using the entire training split. As
an example, consider the training of the shared-private multitask network for depression level regression and emotion
intensity regression: SP MT. DLR+EIR. The DLR-specific
LSTM layer, the shared-LSTM layer, and the corresponding
attention fusion network and fully-connected network are
trained for N DLR epochs without updating the weights of the
EIR-specific layers. For the next N EIR epochs, the EIR-specific
LSTM layer, the shared-LSTM layer, and the corresponding
attention fusion network and fully-connected network are
trained without updating the weights of the DLR-specific layers. Here, N DLR and N EIR are treated as hyperparameters. We
go on training the network in this alternating fashion till a
maximum number of iterations N total (the total number of
times the shared-LSTM layer is trained) is reached. The model
that shows best performance on the development split over all
iterations is chosen for testing. The pseudo-code for our training procedure is shown in algorithm 1.
Algorithm 1 FS/SP/ASP MT. T1 + T2 training.
1: n total ! 1
2: while n total 1 N total do
3:
for n T1 ! 1 to N T1 do
4:
Update T1 -specific and Shared weights
n total ! n total + 1
5:
6:
for n T2 ! 1 to N T2 do
7:
Update T2 -specific and Shared weights
n total ! n total + 1
8:
The architectures have been implemented with Keras11 and
hyperparameters have been optimized through grid search.
Note that all learning models are trained on the basis of stratified 5-cross validation, thus keeping the data distribution
between training, development and test datasets. In particular,
the best of the 5 models over the development set is applied to
classify/regress the examples in the test set.
5. Results
In order to test our hypothesis, we perform a series of experiments for three different tasks: (1) Depression Level Regression
(DLR), which aims to assign a value between 0 to 24 (that is,
the PHQ-8 score) to a given patient interview transcript, (2)
Depression Level Classification (DLC), whose objective is to
identify the correct discrete class of depression level (Noneminimal, Mild, Moderate, Moderately severe, Severe), and (3)
Emotion Intensity Regression (EIR), which regresses a [0-3]
value for each of the six Ekman emotions (happiness, sadness,
anger, fear, disgust and surprise) for a given user transcript.
Five different models serve as baselines. That is, each task is
first modeled as a single-task problem, and two unaware-emotion multi-task (fully-shared and shared-private) architectures
are implemented that combine both DLR and DLC.12 Three
different combinations of emotion-aware multi-task frameworks are tested, for each one of the three theoretical models
(fully-shared, shared-private and adversarial shared-private): (1)
DLC combined with EIR, (2) DLR combined with EIR, and
(3) DLC combined with both DLR and EIR.13
To evaluate regression/classification results, we use wellknown evaluation metrics that are standard for depression level
estimation [40]: (1) Accuracy, F1 score and Matthews Correlation Coefficient (MCC) for classification; (2) Root Mean
Square Error (RMSE), Mean Average Error (MAE), Coefficient of Determination (R2) and Symmetric Mean Absolute
Percentage Error (SMAPE) for regression. In particular, we
include two other metrics (Over and Under), that complement
Accuracy and evaluate how much a learning model over-evaluates (Over) or under-evaluates (Under) the correct result.
Such metrics are important to understand the behavior of
learning models. But, as far as we know, they are not presented in related works. For classification, Accuracy, Over and
Under sum to 100% and are defined in equations 2 and 3. For
11
https://keras.io.
These baselines correspond to the 5 first rows of Table 2.
These models correspond to the 9 last rows of Table 2.
12
13
AUGUST 2020 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE
55
https://keras.io/
IEEE Computational Intelligence Magazine - August 2020
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2020
Contents
IEEE Computational Intelligence Magazine - August 2020 - Cover1
IEEE Computational Intelligence Magazine - August 2020 - Cover2
IEEE Computational Intelligence Magazine - August 2020 - Contents
IEEE Computational Intelligence Magazine - August 2020 - 2
IEEE Computational Intelligence Magazine - August 2020 - 3
IEEE Computational Intelligence Magazine - August 2020 - 4
IEEE Computational Intelligence Magazine - August 2020 - 5
IEEE Computational Intelligence Magazine - August 2020 - 6
IEEE Computational Intelligence Magazine - August 2020 - 7
IEEE Computational Intelligence Magazine - August 2020 - 8
IEEE Computational Intelligence Magazine - August 2020 - 9
IEEE Computational Intelligence Magazine - August 2020 - 10
IEEE Computational Intelligence Magazine - August 2020 - 11
IEEE Computational Intelligence Magazine - August 2020 - 12
IEEE Computational Intelligence Magazine - August 2020 - 13
IEEE Computational Intelligence Magazine - August 2020 - 14
IEEE Computational Intelligence Magazine - August 2020 - 15
IEEE Computational Intelligence Magazine - August 2020 - 16
IEEE Computational Intelligence Magazine - August 2020 - 17
IEEE Computational Intelligence Magazine - August 2020 - 18
IEEE Computational Intelligence Magazine - August 2020 - 19
IEEE Computational Intelligence Magazine - August 2020 - 20
IEEE Computational Intelligence Magazine - August 2020 - 21
IEEE Computational Intelligence Magazine - August 2020 - 22
IEEE Computational Intelligence Magazine - August 2020 - 23
IEEE Computational Intelligence Magazine - August 2020 - 24
IEEE Computational Intelligence Magazine - August 2020 - 25
IEEE Computational Intelligence Magazine - August 2020 - 26
IEEE Computational Intelligence Magazine - August 2020 - 27
IEEE Computational Intelligence Magazine - August 2020 - 28
IEEE Computational Intelligence Magazine - August 2020 - 29
IEEE Computational Intelligence Magazine - August 2020 - 30
IEEE Computational Intelligence Magazine - August 2020 - 31
IEEE Computational Intelligence Magazine - August 2020 - 32
IEEE Computational Intelligence Magazine - August 2020 - 33
IEEE Computational Intelligence Magazine - August 2020 - 34
IEEE Computational Intelligence Magazine - August 2020 - 35
IEEE Computational Intelligence Magazine - August 2020 - 36
IEEE Computational Intelligence Magazine - August 2020 - 37
IEEE Computational Intelligence Magazine - August 2020 - 38
IEEE Computational Intelligence Magazine - August 2020 - 39
IEEE Computational Intelligence Magazine - August 2020 - 40
IEEE Computational Intelligence Magazine - August 2020 - 41
IEEE Computational Intelligence Magazine - August 2020 - 42
IEEE Computational Intelligence Magazine - August 2020 - 43
IEEE Computational Intelligence Magazine - August 2020 - 44
IEEE Computational Intelligence Magazine - August 2020 - 45
IEEE Computational Intelligence Magazine - August 2020 - 46
IEEE Computational Intelligence Magazine - August 2020 - 47
IEEE Computational Intelligence Magazine - August 2020 - 48
IEEE Computational Intelligence Magazine - August 2020 - 49
IEEE Computational Intelligence Magazine - August 2020 - 50
IEEE Computational Intelligence Magazine - August 2020 - 51
IEEE Computational Intelligence Magazine - August 2020 - 52
IEEE Computational Intelligence Magazine - August 2020 - 53
IEEE Computational Intelligence Magazine - August 2020 - 54
IEEE Computational Intelligence Magazine - August 2020 - 55
IEEE Computational Intelligence Magazine - August 2020 - 56
IEEE Computational Intelligence Magazine - August 2020 - 57
IEEE Computational Intelligence Magazine - August 2020 - 58
IEEE Computational Intelligence Magazine - August 2020 - 59
IEEE Computational Intelligence Magazine - August 2020 - 60
IEEE Computational Intelligence Magazine - August 2020 - 61
IEEE Computational Intelligence Magazine - August 2020 - 62
IEEE Computational Intelligence Magazine - August 2020 - 63
IEEE Computational Intelligence Magazine - August 2020 - 64
IEEE Computational Intelligence Magazine - August 2020 - 65
IEEE Computational Intelligence Magazine - August 2020 - 66
IEEE Computational Intelligence Magazine - August 2020 - 67
IEEE Computational Intelligence Magazine - August 2020 - 68
IEEE Computational Intelligence Magazine - August 2020 - 69
IEEE Computational Intelligence Magazine - August 2020 - 70
IEEE Computational Intelligence Magazine - August 2020 - 71
IEEE Computational Intelligence Magazine - August 2020 - 72
IEEE Computational Intelligence Magazine - August 2020 - 73
IEEE Computational Intelligence Magazine - August 2020 - 74
IEEE Computational Intelligence Magazine - August 2020 - 75
IEEE Computational Intelligence Magazine - August 2020 - 76
IEEE Computational Intelligence Magazine - August 2020 - Cover3
IEEE Computational Intelligence Magazine - August 2020 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com