IEEE Computational Intelligence Magazine - February 2022 - 71
model would be dependent on policy learning
model used. Finally, a different contribution
ranking scheme than the one presented in
Section IV could also be proposed (i.e.,
accounting for more complex objective metrics
to better highlight the order of importance
in the team). For instance, each set of observations
per agent could be ranked in order of
quality or average didactic importance in order
to assess the learning agents more fairly, with respect to the
quality of the data they were exposed to.
Shapley values could be a good way to detect
biases in the learning of a policy, since they require
analyzing the individual behavior of each agent
and this could highlight disparities between their
different strategies and abilities.
Acknowledgments
We thank Frédéric Herbreteau, Adrien Bennetot and Léo Heidelberger
for their help and support. N. Díaz-Rodríguez is
currently supported by the Spanish Government Juan de la
Cierva Incorporación contract (IJC2019-039152-I).
References
[1] V. Mnih et al., " Asynchronous methods for deep reinforcement learning, " in Proc. Int.
Conf. Mach. Learn., PMLR, 2016, pp. 1928-1937.
[2] L. Espeholt et al., " Impala: Scalable distributed deep-RL with importance weighted
actor-learner architectures, " in Proc. Int. Conf. Mach. Learn., PMLR, 2018, pp. 1407-1416.
[3] M. T. Ribeiro, S. Singh, and C. Guestrin, " Why should I trust you?' Explaining the
predictions of any classifier, " in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data
Mining, 2016, pp. 1135-1144.
[4] S. El-Sappagh, J. M. Alonso, S. R. Islam, A. M. Sultan, and K. S. Kwak, " A multilayer
multimodal detection and prediction model based on explainable artificial intelligence
for Alzheimer's disease, " Scientific Rep., vol. 11, no. 1, pp. 1-26, 2021, doi: 10.1038/
s41598-021-82098-3.
[5] U. Schlegel, H. Arnout, M. El-Assady, D. Oelke, and D. A. Keim, " Towards a rigorous
evaluation of XAI methods on time series, " in Proc. IEEE/CVF Int. Conf. Comput. Vision
Workshop (ICCVW), 2019, pp. 4197-4201.
[6] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez, " Explainability in deep reinforcement
learning, " Knowl.-Based Syst., vol. 214, p. 106,685, 2021. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0950705120308145, doi: 10.1016/j.
knosys.2020.106685.
[7] E. Puiutta and E. M. Veith, " Explainable reinforcement learning: A survey, " in International
Cross-Domain Conference for Machine Learning and Knowledge Extraction. SpringerVerlag,
2020, pp. 77-95.
[8] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere, " Explainable reinforcement
learning through a causal lens, " in Proc. AAAI Conf. Artif. Intell., 2020, vol. 34, no. 3, pp.
2493-2500, doi: 10.1609/aaai.v34i03.5631.
[9] S. Greydanus, A. Koul, J. Dodge, and A. Fern, " Visualizing and understanding Atari
agents, " in Proc. Int. Conf. Mach. Learn., PMLR, 2018, pp. 1792-1801.
[10] S. M. Lundberg and S.-I. Lee, " A unified approach to interpreting model predictions, "
in Proc. 31st Int. Conf. Neural Inf. Process. Syst., 2017, pp. 4768-4777.
[11] L. Shapley, " A value for N-person games, " in Contributions Theory Games, 1953.
[12] A. Andres, E. Villar-Rodriguez, A. D. Martinez, and J. D. Ser, " Collaborative exploration
and reinforcement learning between heterogeneously skilled agents in environments
with sparse rewards, " in Proc. 2021 Int. Joint Conf. Neural Netw. (IJCNN), pp. 1-10,
doi: 10.1109/IJCNN52387.2021.9534146.
[13] G. Hardin, " The tragedy of the commons, " J. Natural Res. Policy Res., vol. 1, no. 3,
pp. 243-253, 2009, doi:10.1080/19390450903037302.
[14] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, " Multi-agent actorcritic
for mixed cooperative-competitive environments, " in Proc. Neural Inf. Process. Syst.
(NIPS), 2017.
[15] J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel, " Multi-agent
reinforcement learning in sequential social dilemmas, " in Proc. 16th Conf. Auton. Agents
MultiAgent Syst. AAMAS '17, 2017, pp. 464-473.
[16] A. B. Arrieta et al., " Explainable artificial intelligence (XAI): Concepts, taxonomies,
opportunities and challenges toward responsible AI, " Inf. Fusion, vol. 58, pp. 82-115,
2020, doi: 10.1016/j.inffus.2019.12.012.
[17] K. K. Ndousse, D. Eck, S. Levine, and N. Jaques, " Emergent social learning via multiagent
reinforcement learning, " in Proc. Int. Conf. Mach. Learn., PMLR, 2021, pp. 7991-8004.
[18] J. Perolat, J. Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel, " A multiagent
reinforcement learning model of common-pool resource appropriation, " in Proc.
31st Int. Conf. Neural Inf. Process. Syst., 2017, pp. 3646-3655.
[19] E. Hughes et al., " Inequity aversion improves cooperation in intertemporal social
dilemmas, " in Proc. Adv. Neural Inf. Process. Syst. 31, 2018, vol. 31, pp. 1-11.
[20] M. Staniak and P. Biecek, " Explanations of model predictions with live and
breakDown packages, " R J., vol. 10, no. 2, pp. 395-409, 2018, doi: 10.32614/RJ2018-072.
[21]
P. Giudici and E. Raffinetti, " Shapley-Lorenz eXplainable artificial intelligence, "
Expert Syst. Appl., vol. 167, p. 114,104, 2021, doi: 10.1016/j.eswa.2020.114104.
[22] D. Shim, Z. Mai, J. Jeong, S. Sanner, H. Kim, and J. Jang, " Online class-incremental
continual learning with adversarial Shapley value, " in Proc. AAAI Conf. Artif. Intell., 2021,
vol. 35, no. 11, pp. 9630-9638.
[23] M. Sundararajan and A. Najmi, " The many Shapley values for model explanation, " in
Proc. Int. Conf. Mach. Learn., PMLR, 2020, pp. 9269-9278.
[24] J. Wang, Y. Zhang, T.-K. Kim, and Y. Gu, " Shapley Q-value: A local reward approach
to solve global reward games, " in Proc. AAAI Conf. Artif. Intell., 2020, vol. 34, no.
5, pp. 7285-7292, doi: 10.1609/aaai.v34i05.6220.
[25] B. R. Kiran et al., " Deep reinforcement learning for autonomous driving: A
survey, " IEEE Trans. Intell. Transp. Syst., early access, Feb. 9, 2021, doi: 10.1109/
TITS.2021.3054625.
[26] H. Nguyen and H. La, " Review of deep reinforcement learning for robot manipulation, "
in Proc. 3rd IEEE Int. Conf. Robot. Comput. (IRC), 2019, pp. 590-595.
[27] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez,
" Continual learning for robotics: Definition, framework, learning strategies,
opportunities and challenges, " Inf. Fusion, vol. 58, pp. 52-68, 2020, doi: 10.1016/j.
inffus.2019.12.004.
[28] D. Lee, N. Jaques, J. C. Kew, D. Eck, D. Schuurmans, and A. Faust, " Joint attention
for multi-agent coordination and social learning, " 2021, arXiv:2104.07750.
[29] K. Ndousse, D. Eck, S. Levine, and N. Jaques, " Multi-agent social reinforcement
learning improves generalization, " 2020, arXiv:a2010.
[30] J. Duffy and J. Ochs, " Cooperative behavior and the frequency of social interaction, "
Games Econ. Behav., vol. 66, no. 2, pp. 785-812, 2009. [Online]. Available: http://www
.sciencedirect.com/science/article/pii/S089982560800139, doi: 10.1016/j.geb.2008.07.003.
[31] A. M. Colman, " Cooperation, psychological game theory, and limitations of rationality
in social interaction, " Behav. Brain Sci., vol. 26, no. 2, pp. 139-198, 2003. doi:
10.1017/S0140525X03000050.
[32] N. Jaques et al., " Social Influence as Intrinsic Motivation for multi-agent deep reinforcement
learning, " in Proc. Int. Conf. Mach. Learn., PMLR, 2019, pp. 3040-3049.
[33] K. Ndousse, D. Eck, S. Levine, and N. Jaques, " Learning social learning, " 2020,
arxiv:2010.00581.
[34] C. Molnar, " Interpretable machine learning, " 2019. https://christophm.github.io/
interpretable-ml-book/
[35] L. S. Shapley, The Shapley Value: Essays in Honor. Cambridge, U.K.: Cambridge Univ.
Press, 1988.
[36] E. Friedman and H. Moulin, " Three methods to share joint costs or surplus, " J. Econ.
Theory, vol. 87, no. 2, pp. 275-312, 1999. [Online]. Available: http://www.sciencedirect
.com/science/article/pii/S0022053199925346, doi: 10.1006/jeth.1999.2534.
[37] T. Chu, S. Qu, and J. Wang, " Large-scale multi-agent reinforcement learning using
image-based state representation, " in Proc. IEEE 55th Conf. Decis. Contr. (CDC), 2016,
pp. 7592-7597.
[38] U. Faigle and W. Kern, " The Shapley value for cooperative games under precedence
constraints, " Univ. of Twente, Faculty of Math. Sci., no. 1025, 1992.
[39] E. Štrumbelj and I. Kononenko, " Explaining prediction models and individual predictions
with feature contributions, " Knowl. Inf. Syst., vol. 41, no. 3, pp. 647-665, 2014.
[Online]. Available: https://moodle.telekom.ftn.uns.ac.rs/pluginfile.php/13342/mod_
folder/content/0/Feature%20importance%20paper.pdf?forcedownload=1, doi: 10.1007/
s10115-013-0679-x.
[40] A. Tallón-Ballesteros and C. Chen, " Explainable AI: Using Shapley value to explain
complex anomaly detection ML-based systems, " in Machine Learning and Artificial Intelligence,
vol. 332, 2020, p. 152.
[41] T. Lillicrap et al., " Continuous control with deep reinforcement learning, " CoRR,
vol. abs/1509.02971, 2016.
[42] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, " Deep reinforcement
learning that matters, " in Proc. AAAI Conf. Artif. Intell., 2018, vol. 32, no. 1.
[43] E. Vinitsky, N. Jaques, J. Leibo, A. Castenada, and E. Hughes, " An open source
implementation of sequential social dilemma games, " GitHub, 2019. https://github.com/
eugenevinitsky/sequential_ social_ dilemma_ games/issues/182
[44] C. Gini, Variabilità e Mutabilità, reprint in Memorie di metodologica statistica ed. E.
Pizetti, Ed. 1912.
[45] P. J. Huber, Robust Statistics, vol. 523. Hoboken, NJ, USA: Wiley, 2004.
[46] R. A. Maronna, R. D. Martin, V. J. Yohai, and M. Salibián-Barrera, Robust Statistics:
Theory and Methods (with R). Hoboken, NJ, USA: Wiley, 2019.
[47] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel, Robust Statistics:
The Approach based on Influence Functions, vol. 196. Hoboken, NJ, USA: Wiley, 2011.
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 71
http://www.sciencedirect.com/science/article/pii/S089982560800139
http://www.sciencedirect.com/science/article/pii/S089982560800139
http://www.sciencedirect.com/science/article/pii/S0950705120308145
https://christophm.github.io/interpretable-ml-book/
https://christophm.github.io/interpretable-ml-book/
http://www.sciencedirect.com/science/article/pii/S0022053199925346
http://www.sciencedirect.com/science/article/pii/S0022053199925346
https://moodle.telekom.ftn.uns.ac.rs/pluginfile.php/13342/mod_folder/content/0/Feature%20importance%20paper.pdf?forcedownload=1
https://moodle.telekom.ftn.uns.ac.rs/pluginfile.php/13342/mod_folder/content/0/Feature%20importance%20paper.pdf?forcedownload=1
https://www.github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182
https://www.github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182
IEEE Computational Intelligence Magazine - February 2022
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
Contens
IEEE Computational Intelligence Magazine - February 2022 - Cover1
IEEE Computational Intelligence Magazine - February 2022 - Cover2
IEEE Computational Intelligence Magazine - February 2022 - Contens
IEEE Computational Intelligence Magazine - February 2022 - 2
IEEE Computational Intelligence Magazine - February 2022 - 3
IEEE Computational Intelligence Magazine - February 2022 - 4
IEEE Computational Intelligence Magazine - February 2022 - 5
IEEE Computational Intelligence Magazine - February 2022 - 6
IEEE Computational Intelligence Magazine - February 2022 - 7
IEEE Computational Intelligence Magazine - February 2022 - 8
IEEE Computational Intelligence Magazine - February 2022 - 9
IEEE Computational Intelligence Magazine - February 2022 - 10
IEEE Computational Intelligence Magazine - February 2022 - 11
IEEE Computational Intelligence Magazine - February 2022 - 12
IEEE Computational Intelligence Magazine - February 2022 - 13
IEEE Computational Intelligence Magazine - February 2022 - 14
IEEE Computational Intelligence Magazine - February 2022 - 15
IEEE Computational Intelligence Magazine - February 2022 - 16
IEEE Computational Intelligence Magazine - February 2022 - 17
IEEE Computational Intelligence Magazine - February 2022 - 18
IEEE Computational Intelligence Magazine - February 2022 - 19
IEEE Computational Intelligence Magazine - February 2022 - 20
IEEE Computational Intelligence Magazine - February 2022 - 21
IEEE Computational Intelligence Magazine - February 2022 - 22
IEEE Computational Intelligence Magazine - February 2022 - 23
IEEE Computational Intelligence Magazine - February 2022 - 24
IEEE Computational Intelligence Magazine - February 2022 - 25
IEEE Computational Intelligence Magazine - February 2022 - 26
IEEE Computational Intelligence Magazine - February 2022 - 27
IEEE Computational Intelligence Magazine - February 2022 - 28
IEEE Computational Intelligence Magazine - February 2022 - 29
IEEE Computational Intelligence Magazine - February 2022 - 30
IEEE Computational Intelligence Magazine - February 2022 - 31
IEEE Computational Intelligence Magazine - February 2022 - 32
IEEE Computational Intelligence Magazine - February 2022 - 33
IEEE Computational Intelligence Magazine - February 2022 - 34
IEEE Computational Intelligence Magazine - February 2022 - 35
IEEE Computational Intelligence Magazine - February 2022 - 36
IEEE Computational Intelligence Magazine - February 2022 - 37
IEEE Computational Intelligence Magazine - February 2022 - 38
IEEE Computational Intelligence Magazine - February 2022 - 39
IEEE Computational Intelligence Magazine - February 2022 - 40
IEEE Computational Intelligence Magazine - February 2022 - 41
IEEE Computational Intelligence Magazine - February 2022 - 42
IEEE Computational Intelligence Magazine - February 2022 - 43
IEEE Computational Intelligence Magazine - February 2022 - 44
IEEE Computational Intelligence Magazine - February 2022 - 45
IEEE Computational Intelligence Magazine - February 2022 - 46
IEEE Computational Intelligence Magazine - February 2022 - 47
IEEE Computational Intelligence Magazine - February 2022 - 48
IEEE Computational Intelligence Magazine - February 2022 - 49
IEEE Computational Intelligence Magazine - February 2022 - 50
IEEE Computational Intelligence Magazine - February 2022 - 51
IEEE Computational Intelligence Magazine - February 2022 - 52
IEEE Computational Intelligence Magazine - February 2022 - 53
IEEE Computational Intelligence Magazine - February 2022 - 54
IEEE Computational Intelligence Magazine - February 2022 - 55
IEEE Computational Intelligence Magazine - February 2022 - 56
IEEE Computational Intelligence Magazine - February 2022 - 57
IEEE Computational Intelligence Magazine - February 2022 - 58
IEEE Computational Intelligence Magazine - February 2022 - 59
IEEE Computational Intelligence Magazine - February 2022 - 60
IEEE Computational Intelligence Magazine - February 2022 - 61
IEEE Computational Intelligence Magazine - February 2022 - 62
IEEE Computational Intelligence Magazine - February 2022 - 63
IEEE Computational Intelligence Magazine - February 2022 - 64
IEEE Computational Intelligence Magazine - February 2022 - 65
IEEE Computational Intelligence Magazine - February 2022 - 66
IEEE Computational Intelligence Magazine - February 2022 - 67
IEEE Computational Intelligence Magazine - February 2022 - 68
IEEE Computational Intelligence Magazine - February 2022 - 69
IEEE Computational Intelligence Magazine - February 2022 - 70
IEEE Computational Intelligence Magazine - February 2022 - 71
IEEE Computational Intelligence Magazine - February 2022 - 72
IEEE Computational Intelligence Magazine - February 2022 - 73
IEEE Computational Intelligence Magazine - February 2022 - 74
IEEE Computational Intelligence Magazine - February 2022 - 75
IEEE Computational Intelligence Magazine - February 2022 - 76
IEEE Computational Intelligence Magazine - February 2022 - 77
IEEE Computational Intelligence Magazine - February 2022 - 78
IEEE Computational Intelligence Magazine - February 2022 - 79
IEEE Computational Intelligence Magazine - February 2022 - 80
IEEE Computational Intelligence Magazine - February 2022 - 81
IEEE Computational Intelligence Magazine - February 2022 - 82
IEEE Computational Intelligence Magazine - February 2022 - 83
IEEE Computational Intelligence Magazine - February 2022 - 84
IEEE Computational Intelligence Magazine - February 2022 - 85
IEEE Computational Intelligence Magazine - February 2022 - 86
IEEE Computational Intelligence Magazine - February 2022 - 87
IEEE Computational Intelligence Magazine - February 2022 - 88
IEEE Computational Intelligence Magazine - February 2022 - 89
IEEE Computational Intelligence Magazine - February 2022 - 90
IEEE Computational Intelligence Magazine - February 2022 - 91
IEEE Computational Intelligence Magazine - February 2022 - 92
IEEE Computational Intelligence Magazine - February 2022 - 93
IEEE Computational Intelligence Magazine - February 2022 - 94
IEEE Computational Intelligence Magazine - February 2022 - 95
IEEE Computational Intelligence Magazine - February 2022 - 96
IEEE Computational Intelligence Magazine - February 2022 - 97
IEEE Computational Intelligence Magazine - February 2022 - 98
IEEE Computational Intelligence Magazine - February 2022 - 99
IEEE Computational Intelligence Magazine - February 2022 - 100
IEEE Computational Intelligence Magazine - February 2022 - 101
IEEE Computational Intelligence Magazine - February 2022 - 102
IEEE Computational Intelligence Magazine - February 2022 - 103
IEEE Computational Intelligence Magazine - February 2022 - 104
IEEE Computational Intelligence Magazine - February 2022 - 105
IEEE Computational Intelligence Magazine - February 2022 - 106
IEEE Computational Intelligence Magazine - February 2022 - 107
IEEE Computational Intelligence Magazine - February 2022 - 108
IEEE Computational Intelligence Magazine - February 2022 - 109
IEEE Computational Intelligence Magazine - February 2022 - 110
IEEE Computational Intelligence Magazine - February 2022 - 111
IEEE Computational Intelligence Magazine - February 2022 - 112
IEEE Computational Intelligence Magazine - February 2022 - 113
IEEE Computational Intelligence Magazine - February 2022 - 114
IEEE Computational Intelligence Magazine - February 2022 - 115
IEEE Computational Intelligence Magazine - February 2022 - 116
IEEE Computational Intelligence Magazine - February 2022 - Cover3
IEEE Computational Intelligence Magazine - February 2022 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com