IEEE Computational Intelligence Magazine - February 2022 - 71

model would be dependent on policy learning
model used. Finally, a different contribution
ranking scheme than the one presented in
Section IV could also be proposed (i.e.,
accounting for more complex objective metrics
to better highlight the order of importance
in the team). For instance, each set of observations
per agent could be ranked in order of
quality or average didactic importance in order
to assess the learning agents more fairly, with respect to the
quality of the data they were exposed to.
Shapley values could be a good way to detect
biases in the learning of a policy, since they require
analyzing the individual behavior of each agent
and this could highlight disparities between their
different strategies and abilities.
Acknowledgments
We thank Frédéric Herbreteau, Adrien Bennetot and Léo Heidelberger
for their help and support. N. Díaz-Rodríguez is
currently supported by the Spanish Government Juan de la
Cierva Incorporación contract (IJC2019-039152-I).
References
[1] V. Mnih et al., " Asynchronous methods for deep reinforcement learning, " in Proc. Int.
Conf. Mach. Learn., PMLR, 2016, pp. 1928-1937.
[2] L. Espeholt et al., " Impala: Scalable distributed deep-RL with importance weighted
actor-learner architectures, " in Proc. Int. Conf. Mach. Learn., PMLR, 2018, pp. 1407-1416.
[3] M. T. Ribeiro, S. Singh, and C. Guestrin, " Why should I trust you?' Explaining the
predictions of any classifier, " in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data
Mining, 2016, pp. 1135-1144.
[4] S. El-Sappagh, J. M. Alonso, S. R. Islam, A. M. Sultan, and K. S. Kwak, " A multilayer
multimodal detection and prediction model based on explainable artificial intelligence
for Alzheimer's disease, " Scientific Rep., vol. 11, no. 1, pp. 1-26, 2021, doi: 10.1038/
s41598-021-82098-3.
[5] U. Schlegel, H. Arnout, M. El-Assady, D. Oelke, and D. A. Keim, " Towards a rigorous
evaluation of XAI methods on time series, " in Proc. IEEE/CVF Int. Conf. Comput. Vision
Workshop (ICCVW), 2019, pp. 4197-4201.
[6] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez, " Explainability in deep reinforcement
learning, " Knowl.-Based Syst., vol. 214, p. 106,685, 2021. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0950705120308145, doi: 10.1016/j.
knosys.2020.106685.
[7] E. Puiutta and E. M. Veith, " Explainable reinforcement learning: A survey, " in International
Cross-Domain Conference for Machine Learning and Knowledge Extraction. SpringerVerlag,
2020, pp. 77-95.
[8] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere, " Explainable reinforcement
learning through a causal lens, " in Proc. AAAI Conf. Artif. Intell., 2020, vol. 34, no. 3, pp.
2493-2500, doi: 10.1609/aaai.v34i03.5631.
[9] S. Greydanus, A. Koul, J. Dodge, and A. Fern, " Visualizing and understanding Atari
agents, " in Proc. Int. Conf. Mach. Learn., PMLR, 2018, pp. 1792-1801.
[10] S. M. Lundberg and S.-I. Lee, " A unified approach to interpreting model predictions, "
in Proc. 31st Int. Conf. Neural Inf. Process. Syst., 2017, pp. 4768-4777.
[11] L. Shapley, " A value for N-person games, " in Contributions Theory Games, 1953.
[12] A. Andres, E. Villar-Rodriguez, A. D. Martinez, and J. D. Ser, " Collaborative exploration
and reinforcement learning between heterogeneously skilled agents in environments
with sparse rewards, " in Proc. 2021 Int. Joint Conf. Neural Netw. (IJCNN), pp. 1-10,
doi: 10.1109/IJCNN52387.2021.9534146.
[13] G. Hardin, " The tragedy of the commons, " J. Natural Res. Policy Res., vol. 1, no. 3,
pp. 243-253, 2009, doi:10.1080/19390450903037302.
[14] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, " Multi-agent actorcritic
for mixed cooperative-competitive environments, " in Proc. Neural Inf. Process. Syst.
(NIPS), 2017.
[15] J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel, " Multi-agent
reinforcement learning in sequential social dilemmas, " in Proc. 16th Conf. Auton. Agents
MultiAgent Syst. AAMAS '17, 2017, pp. 464-473.
[16] A. B. Arrieta et al., " Explainable artificial intelligence (XAI): Concepts, taxonomies,
opportunities and challenges toward responsible AI, " Inf. Fusion, vol. 58, pp. 82-115,
2020, doi: 10.1016/j.inffus.2019.12.012.
[17] K. K. Ndousse, D. Eck, S. Levine, and N. Jaques, " Emergent social learning via multiagent
reinforcement learning, " in Proc. Int. Conf. Mach. Learn., PMLR, 2021, pp. 7991-8004.
[18] J. Perolat, J. Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel, " A multiagent
reinforcement learning model of common-pool resource appropriation, " in Proc.
31st Int. Conf. Neural Inf. Process. Syst., 2017, pp. 3646-3655.
[19] E. Hughes et al., " Inequity aversion improves cooperation in intertemporal social
dilemmas, " in Proc. Adv. Neural Inf. Process. Syst. 31, 2018, vol. 31, pp. 1-11.
[20] M. Staniak and P. Biecek, " Explanations of model predictions with live and
breakDown packages, " R J., vol. 10, no. 2, pp. 395-409, 2018, doi: 10.32614/RJ2018-072.
[21]
P. Giudici and E. Raffinetti, " Shapley-Lorenz eXplainable artificial intelligence, "
Expert Syst. Appl., vol. 167, p. 114,104, 2021, doi: 10.1016/j.eswa.2020.114104.
[22] D. Shim, Z. Mai, J. Jeong, S. Sanner, H. Kim, and J. Jang, " Online class-incremental
continual learning with adversarial Shapley value, " in Proc. AAAI Conf. Artif. Intell., 2021,
vol. 35, no. 11, pp. 9630-9638.
[23] M. Sundararajan and A. Najmi, " The many Shapley values for model explanation, " in
Proc. Int. Conf. Mach. Learn., PMLR, 2020, pp. 9269-9278.
[24] J. Wang, Y. Zhang, T.-K. Kim, and Y. Gu, " Shapley Q-value: A local reward approach
to solve global reward games, " in Proc. AAAI Conf. Artif. Intell., 2020, vol. 34, no.
5, pp. 7285-7292, doi: 10.1609/aaai.v34i05.6220.
[25] B. R. Kiran et al., " Deep reinforcement learning for autonomous driving: A
survey, " IEEE Trans. Intell. Transp. Syst., early access, Feb. 9, 2021, doi: 10.1109/
TITS.2021.3054625.
[26] H. Nguyen and H. La, " Review of deep reinforcement learning for robot manipulation, "
in Proc. 3rd IEEE Int. Conf. Robot. Comput. (IRC), 2019, pp. 590-595.
[27] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez,
" Continual learning for robotics: Definition, framework, learning strategies,
opportunities and challenges, " Inf. Fusion, vol. 58, pp. 52-68, 2020, doi: 10.1016/j.
inffus.2019.12.004.
[28] D. Lee, N. Jaques, J. C. Kew, D. Eck, D. Schuurmans, and A. Faust, " Joint attention
for multi-agent coordination and social learning, " 2021, arXiv:2104.07750.
[29] K. Ndousse, D. Eck, S. Levine, and N. Jaques, " Multi-agent social reinforcement
learning improves generalization, " 2020, arXiv:a2010.
[30] J. Duffy and J. Ochs, " Cooperative behavior and the frequency of social interaction, "
Games Econ. Behav., vol. 66, no. 2, pp. 785-812, 2009. [Online]. Available: http://www
.sciencedirect.com/science/article/pii/S089982560800139, doi: 10.1016/j.geb.2008.07.003.
[31] A. M. Colman, " Cooperation, psychological game theory, and limitations of rationality
in social interaction, " Behav. Brain Sci., vol. 26, no. 2, pp. 139-198, 2003. doi:
10.1017/S0140525X03000050.
[32] N. Jaques et al., " Social Influence as Intrinsic Motivation for multi-agent deep reinforcement
learning, " in Proc. Int. Conf. Mach. Learn., PMLR, 2019, pp. 3040-3049.
[33] K. Ndousse, D. Eck, S. Levine, and N. Jaques, " Learning social learning, " 2020,
arxiv:2010.00581.
[34] C. Molnar, " Interpretable machine learning, " 2019. https://christophm.github.io/
interpretable-ml-book/
[35] L. S. Shapley, The Shapley Value: Essays in Honor. Cambridge, U.K.: Cambridge Univ.
Press, 1988.
[36] E. Friedman and H. Moulin, " Three methods to share joint costs or surplus, " J. Econ.
Theory, vol. 87, no. 2, pp. 275-312, 1999. [Online]. Available: http://www.sciencedirect
.com/science/article/pii/S0022053199925346, doi: 10.1006/jeth.1999.2534.
[37] T. Chu, S. Qu, and J. Wang, " Large-scale multi-agent reinforcement learning using
image-based state representation, " in Proc. IEEE 55th Conf. Decis. Contr. (CDC), 2016,
pp. 7592-7597.
[38] U. Faigle and W. Kern, " The Shapley value for cooperative games under precedence
constraints, " Univ. of Twente, Faculty of Math. Sci., no. 1025, 1992.
[39] E. Štrumbelj and I. Kononenko, " Explaining prediction models and individual predictions
with feature contributions, " Knowl. Inf. Syst., vol. 41, no. 3, pp. 647-665, 2014.
[Online]. Available: https://moodle.telekom.ftn.uns.ac.rs/pluginfile.php/13342/mod_
folder/content/0/Feature%20importance%20paper.pdf?forcedownload=1, doi: 10.1007/
s10115-013-0679-x.
[40] A. Tallón-Ballesteros and C. Chen, " Explainable AI: Using Shapley value to explain
complex anomaly detection ML-based systems, " in Machine Learning and Artificial Intelligence,
vol. 332, 2020, p. 152.
[41] T. Lillicrap et al., " Continuous control with deep reinforcement learning, " CoRR,
vol. abs/1509.02971, 2016.
[42] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, " Deep reinforcement
learning that matters, " in Proc. AAAI Conf. Artif. Intell., 2018, vol. 32, no. 1.
[43] E. Vinitsky, N. Jaques, J. Leibo, A. Castenada, and E. Hughes, " An open source
implementation of sequential social dilemma games, " GitHub, 2019. https://github.com/
eugenevinitsky/sequential_ social_ dilemma_ games/issues/182
[44] C. Gini, Variabilità e Mutabilità, reprint in Memorie di metodologica statistica ed. E.
Pizetti, Ed. 1912.
[45] P. J. Huber, Robust Statistics, vol. 523. Hoboken, NJ, USA: Wiley, 2004.
[46] R. A. Maronna, R. D. Martin, V. J. Yohai, and M. Salibián-Barrera, Robust Statistics:
Theory and Methods (with R). Hoboken, NJ, USA: Wiley, 2019.
[47] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel, Robust Statistics:
The Approach based on Influence Functions, vol. 196. Hoboken, NJ, USA: Wiley, 2011.
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 71
http://www.sciencedirect.com/science/article/pii/S089982560800139 http://www.sciencedirect.com/science/article/pii/S089982560800139 http://www.sciencedirect.com/science/article/pii/S0950705120308145 https://christophm.github.io/interpretable-ml-book/ https://christophm.github.io/interpretable-ml-book/ http://www.sciencedirect.com/science/article/pii/S0022053199925346 http://www.sciencedirect.com/science/article/pii/S0022053199925346 https://moodle.telekom.ftn.uns.ac.rs/pluginfile.php/13342/mod_folder/content/0/Feature%20importance%20paper.pdf?forcedownload=1 https://moodle.telekom.ftn.uns.ac.rs/pluginfile.php/13342/mod_folder/content/0/Feature%20importance%20paper.pdf?forcedownload=1 https://www.github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182 https://www.github.com/eugenevinitsky/sequential_social_dilemma_games/issues/182
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
