IEEE Computational Intelligence Magazine - August 2020 - 59

emotion-aware and emotion-unaware multi-task architectures
over combinations of three tasks: depression level classification,
depression level regression and emotion intensity regression.
Strong evaluation including new metrics and class-wise results
shows that emotion-aware models outperform emotion-unaware
baselines in a vast majority of tested situations over the standard
benchmarks DAIC-WOZ and CMU-MOSEI. We anticipate
that our work will help to reduce the number of cases of late
treatment of depression, as one can always get an estimate of his/
her PHQ-8 score, without needing to consult a psychiatrist,
especially considering the stigma surrounding this illness. However, current results are not accurate enough to help the therapist
in his diagnosis as model performance is still too low.This should
be a great motivation for future work in depression level estimation. Such research directions include (1) the combination of
text, visual and acoustic modalities following the ideas of [10],
[17], [33], (2) the study of different concurrent tasks for depression estimation and (3) the creation of larger datasets to better
evaluate depression models in terms of class-wise results, that
may also include new biomarkers or descriptors.
Acknowledgments

Dr. Sriparna Saha gratefully acknowledges the Young Faculty
Research Fellowship (YFRF) Award, supported by Visvesvaraya
PhD scheme for Electronics and IT, Ministry of Electronics
and Information Technology (MeitY), Government of India,
being implemented by Digital India Corporation (formerly
Media Lab Asia) for carrying out this research.
References

[1] T. Vos, R. M. Barber, B. Bell, and A. Bertozzi-Villa, "Global, regional, and national
incidence, prevalence, and years lived with disability for 301 acute and chronic diseases
and injuries in 188 countries, 1990-2013: A systematic analysis for the Global Burden of
Disease Study 2013," Lancet, vol. 386, no. 9995, pp. 743-800, Aug. 2015. doi: 10.1016/
S0140-6736(15)60692-4.
[2] A. Ferrari et al., "Burden of depressive disorders by country, sex, age, and year:
Findings from the Global Burden of Disease Study 2010," PLoS Med., vol. 10, no. 11, p.
e1001547, Nov. 2013. doi: 10.1371/journal.pmed.1001547.
[3] A. Beck and B. Alford, Depression: Causes and Treatment. Philadelphia: Univ. of Pennsylvania Press, 2009.
[4] K. Smith, P. Renshaw, and J. Bilelloa, "The diagnosis of depression: Current and emerging methods," Compr. Psychiatry, vol. 54, no. 1, pp. 1-6, 2013. doi: 10.1016/j.comppsych.
2012.06.006.
[5] K. Kroenke, T. Strine, R. Spitzer, J. Williams, J. T. Berry, and A. Mokdad, "The
PHQ-8 as a measure of current depression in the general population," J. Affect. Disord.,
vol. 114, no. 1-3, pp. 163-173, 2008. doi: 10.1016/j.jad.2008.06.026.
[6] S. El-Den, T. Chen, Y.-L. Gan, E. Wong, and C. O'Reilly, "The psychometric properties of depression screening tools in primary healthcare settings: A systematic review," J.
Affect. Disord., vol. 225, pp. 503-522, Jan. 2018. doi: 10.1016/j.jad.2017.08.060.
[7] C. Shin, S.-H. Lee, K.-M. Han, H.-K. Yoon, and C. Han, "Comparison of the usefulness of the PHQ-8 and PHQ-9 for screening for major depressive disorder: Analysis of
psychiatric outpatient data," Psychiatry Investig., vol. 16, no. 4, pp. 300-305, 2019. doi:
10.30773/pi.2019.02.01.
[8] N. Cummins, S. Scherer, J. Krajewski, S. Schnieder, J. Epps, and T. F. Quatieri, "A
review of depression and suicide risk assessment using speech analysis," Speech Commun.,
vol. 71, pp. 10-49, July 2015. doi: 10.1016/j.specom.2015.03.004.
[9] J. T. Wolohan, M. Hiraga, A. Mukherjee, Z. A. Sayyed, and M. Millard, "Detecting linguistic traces of depression in topic-restricted text: Attending to self-stigmatized depression with NLP," in Proc. Int. Workshop Language Cognition and Computational Models, Santa
Fe, Aug. 2018, pp. 11-21.
[10] M. R. Morales, "Multimodal Depression Detection: An Investigation of Features and
Fusion Techniques for Automated Systems," Ph.D. thesis, City Univ. of New York, 2018.
[11] J. Joormann and I. Gotlib, "Emotion regulation in depression: Relation to cognitive inhibition," Cogn. Emot., vol. 24, no. 2, pp. 281-298, 2010. doi: 10.1080/02699930903407948.
[12] R. Thompson, M. Boden, and I. Gotlib, "Emotional variability and clarity in depression and social anxiety," Cogn. Emot., vol. 31, no. 1, pp. 98-108, 2017. doi: 10.1080/02699931.2015.1084908.
[13] P. Ekman and R. Davidson, The Nature of Emotion: Fundamental Questions. London,
U.K.: Oxford Univ. Press, 1994.

[14] P. Liu, X. Qiu, and X. Huang, "Adversarial multi-task learning for text classification," in Proc. Annu. Meeting Association for Computational Linguistics, Vancouver, B.C.,
Canada, July 30-Aug. 4, 2017, pp. 1-10. doi: 10.18653/v1/P17-1001.
[15] S.-A. Qureshi, S. Saha, M. Hasanuzzaman, and G. Dias, "Multitask representation
learning for multimodal estimation of depression level," IEEE Intell. Syst., vol. 34, no. 5,
pp. 45-52, 2019. doi: 10.1109/MIS.2019.2925204.
[16] J. Gratch et al., "The distress analysis interview corpus of human and computer interviews," in Proc. Int. Conf. Language Resources and Evaluation, Reykjavik, May 26-31,
2014, pp. 3123-3128.
[17] A. Zadeh et al., "Multimodal language analysis in the wild: CMU-MOSEI dataset
and interpretable dynamic fusion graph," in Proc. Annu. Meeting Association for Computational Linguistics, Melbourne, July 15-20, 2018, doi: 10.18653/v1/P18-1208.
[18] N. Dewan, J. Luo, and N. Lorenzi, Mental Health Practice in a Digital World: A Clinicians
Guide. New York: Springer-Verlag, 2015.
[19] M. Chatterjee, G. Stratou, S. Scherer, and L.-P. Morency, "Context-based signal
descriptors of heart-rate variability for anxiety assessment," in Proc. IEEE Int. Conf.
Acoustics, Speech and Signal Processing, Florence, Italy, May 4-9, 2014, doi: 10.1109/
ICASSP.2014.6854278.
[20] M. Morales and R. Levitan, "Speech vs. text: A comparative analysis of features for
depression detection systems," in Proc. IEEE Spoken Language Technology Workshop, San
Diego, CA, May 13-16, 2016, doi: 10.1109/SLT.2016.7846256.
[21] D. Hovy, M. Mitchell, and A. Benton, "Multitask learning for mental health conditions with limited social media data," in Proc. Conf. European Chapter Association for Computational Linguistics, Valencia, Spain, Apr. 3-7, 2017, pp. 152-162. doi: 10.18653/v1/
E17-1015.
[22] D. Losada and F. Crestani, "A test collection for research on depression and language
use," in Proc. Int. Conf. Cross-Language Evaluation Forum for European Languages, Évora,
Portugal, Sept. 5-8, 2016, pp. 28-39. doi: 10.1007/978-3-319-44564-9_3.
[23] H. Dibekliog˘lu, Z. Hammal, Y. Yang, and J. Cohn, "Multimodal detection of depression in clinical interviews," in Proc. ACM Int. Conf. Multimodal Interaction, Seattle,
WA, Nov. 9-13, 2015, pp. 307-310. doi: 10.1145/2818346.2820776.
[24] M. Morales, S. Scherer, and R. Levitan, "A linguistically-informed fusion approach
for multimodal depression detection," in Proc. Workshop Computational Linguistics and
Clinical Psychology: From Keyboard Clinic, New Orleans, LA, June 2018, pp. 13-24. doi:
10.18653/v1/W18-0602.
[25] S.-A. Qureshi, M. Hasansuzzaman, S. Saha, and G. Dias, The verbal and non verbal
signals of depression: Combining acoustics, text and visuals for estimating depression
level. Apr. 2019. [Online]. Available: arXiv:1904.07656
[26] B. Sun et al., "A random forest regression method with selected-text feature for depression assessment," in Proc. Annu. Workshop Audio/Visual Emotion Challenge, Mountain
View, CA, Oct. 23-27, 2017, pp. 61-68. doi: 10.1145/3133944.3133951.
[27] J. Bingel and A. Søgaard, "Identifying beneficial task relations for multi-task learning in deep neural networks," in Proc. Conf. European Chapter Association for Computational
Linguistics, Valencia, Spain, Apr. 3-7, 2017, pp. 164-169. doi: 10.18653/v1/E17-2026.
[28] S. Yadav, A. Ekbal, S. Saha, P. Bhattacharyya, and A. Sheth, "Multi-task learning
framework for mining crowd intelligence towards clinical treatment," in Proc. Conf. North
American Chapter Association for Computational Linguistics: Human Language Technologies,
New Orleans, LA, June 1-6, 2018, pp. 271-277. doi: 10.18653/v1/N18-2044.
[29] R. Caruana, "Multitask learning," Mach. Learn., vol. 28, no. 1, pp. 41-75, 1998. doi:
10.1023/A:1007379606734.
[30] D. Cer et al., Universal sentence encoder. Mar. 2018. [Online]. Available: arXiv:1803.11175
[31] S. Hochreiter and J. Schmidhuber, "Long short-term Memory," Neural Comput., vol.
9, no. 8, pp. 1735-1780, 1997. doi: 10.1162/neco.1997.9.8.1735.
[32] T. Young, D. Hazarika, S. Poria, and E. Cambria, Recent trends in deep learning
based natural language processing. Aug. 2017. [Online]. Available: arXiv:1708.02709
[33] S. Poria, E. Cambria, D. Hazarika, N. Mazumder, A. Zadeh, and L.-P. Morency,
"Multi-level multiple attentions for contextual multimodal sentiment analysis," in Proc.
IEEE Int. Conf. Data Mining, New Orleans, LA, Nov. 18-21, 2017, pp. 1033-1038. doi:
10.1109/ICDM.2017.134.
[34] T. Luong, H. Pham, and C. Manning, "Effective approaches to attention-based neural machine translation," in Proc. Conf. Empirical Methods in Natural Language Processing,
Lisbon, Portugal, Sept. 17-21, 2015, pp. 1412-1421. doi: 10.18653/v1/D15-1166.
[35] T. Baltrušaitis, P. Robinson, and L.-P. Morency, "Openface: An open source facial
behavior analysis toolkit," in Proc. IEEE Winter Conf. Applications Computer Vision, Lake
Placid, NY, Mar. 7-9, 2016, pp. 1-10. doi: 10.1109/WACV.2016.7477553.
[36] G. Littlewort et al., "The computer expression recognition toolbox (CERT)," in Proc.
IEEE Int. Conf. and Workshops Automatic Face and Gesture Recognition, Santa Barbara, CA,
Mar. 21-25, 2011, pp. 298-305. doi: 10.1109/FG.2011.5771414.
[37] G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, "COVAREP: A collaborative voice analysis repository for speech technologies," in Proc. IEEE Int. Conf.
Acoustics, Speech and Signal Processing, Florence, Italy, May 4-9, 2014, pp. 960-964. doi:
10.1109/ICASSP.2014.6853739.
[38] K. Kroenke, R. Spitzer, and J. Williams, "The PHQ-9: Validity of a brief depression severity measure," J. Gen. Intern. Med., vol. 16, no. 9, pp. 606-613, 2001. doi:
10.1046/j.1525-1497.2001.016009606.x.
[39] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, MOSI: Multimodal corpus of
sentiment intensity and subjectivity analysis in online opinion videos. June 2016. [Online]. Available: arXiv:1606.06259
[40] F. Ringeval et al., "AVEC 2017: Real-life depression, and affect recognition workshop and challenge," in Proc. Annu. Workshop Audio/Visual Emotion Challenge. Mountain
View, CA, Oct. 23-27, 2017, pp. 3-9. doi: 10.1145/3133944.3133953.

AUGUST 2020 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

59
IEEE Computational Intelligence Magazine - August 2020

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2020
