IEEE Computational Intelligence Magazine - February 2022 - 65
Efficiency U E i =1
= >
Equality E=1
N
/Ri
T
H
(5)
Predator
// RR
i 11
-
==
2NRi
i =1
N
/
Sustainability S EE1 tt tr 0
i
= = / Gi ,where =
N
N 1
i =
Predator 2
Finally, additional experiments were executed on different
Harvest checkpoints to explore how agents cooperate. Each
experiment will be detailed in the subsections below. More
details on the experiments can be found in the Supplementary
Materials.
B. Experiment 1: Agents with Identical
Settings in Multiagent Particle
1) Environment Settings
The goal of this experiment is to answer RQ1 and RQ2
(Section I), i.e., whether the Shapley values of predator
agents correlate with the number of times they catch the
prey, and whether these are a close approximation to the
exact Shapley values.
First, when training a model of three predators using default
settings on the Predator-Prey scenario, one could think that
each predator should provide a similar contribution, as predators
do not have significant differences on their speed, action
space or training method. However, in contrast to this assumption,
statistics in Figure 1 show that the performance of each
predator agent (i.e., the number of times each predator catches
the prey) varies significantly: Predator 3 has a larger contribution
than Predator 1, which has a higher contribution than
Predator 0. In fact, the trained MADDPG model developed a
strategy on which Predator 3 and Predator 2 perform better
than Predator 0 at catching the prey. This can be explained by
the fact that MADDPG provides the same reward to all agents,
instead of only rewarding the highest-contributing agent. This
can bias the training, as explained in [24]. On the contrary,
rewarding only the agent who contributed the most does not
highlight or recognize team strategies where the contribution
of every agent was critical to the shared payout.
2) Shapley Values Analysis
Figure 2 shows Shapley values computed for each agent on
each of the five models, over 1,000 sample episodes per model.
This can be observed in a more convenient way in Figure 1,
which shows Shapley values computed for each agent on a single
model. As hypothesized, the decreasing order of agents'
Shapley values is the following: Predator 2, Predator 1 and
Predator 0. Thus, this first experiment supports RQ1, since
Shapley values are able to correctly map contributions to
agents' usefulness in a cooperative multi-agent setting.
1,000 2,000 3,000 4,000
Number of Times the Prey Get Caught
(a)
Predator 0
[ |] (7)
t
i
2
N N ij
j
(6)
Predator 1
Predator 1
Predator 2
02 46 81012
Shapley Value (Contribution of Each Agent)
(b)
FIGURE 1 Predator-Prey environment: Agents are endowed with the
same speed. Left plot shows predator agents mean performance comparison
out of 10,000 sample episodes over 5 different models (run
2,000 sample episodes each) while right plot presents the Monte
Carlo estimation of Shapley values obtained for each predator agent
(M = 1,000; " random " player exclusion method; mean over 5 models).
Predator 0
20
15
10
5
-5
Method
Noop (MC Estimation)
Random (MC Estimation)
Replace (MC Estimation)
Noop (Real)
Random (Real)
Replace (Real)
FIGURE 2 Predator-Prey environment, same agents' speeds. Comparison
of MC approximation of Shapley values with the exact Shapley
values. Each point relates to one of the 5 different models run.
Predator 1
Predator 2
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 65
Agent Names
Agent Names
Shapley Value
IEEE Computational Intelligence Magazine - February 2022
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
Contens
IEEE Computational Intelligence Magazine - February 2022 - Cover1
IEEE Computational Intelligence Magazine - February 2022 - Cover2
IEEE Computational Intelligence Magazine - February 2022 - Contens
IEEE Computational Intelligence Magazine - February 2022 - 2
IEEE Computational Intelligence Magazine - February 2022 - 3
IEEE Computational Intelligence Magazine - February 2022 - 4
IEEE Computational Intelligence Magazine - February 2022 - 5
IEEE Computational Intelligence Magazine - February 2022 - 6
IEEE Computational Intelligence Magazine - February 2022 - 7
IEEE Computational Intelligence Magazine - February 2022 - 8
IEEE Computational Intelligence Magazine - February 2022 - 9
IEEE Computational Intelligence Magazine - February 2022 - 10
IEEE Computational Intelligence Magazine - February 2022 - 11
IEEE Computational Intelligence Magazine - February 2022 - 12
IEEE Computational Intelligence Magazine - February 2022 - 13
IEEE Computational Intelligence Magazine - February 2022 - 14
IEEE Computational Intelligence Magazine - February 2022 - 15
IEEE Computational Intelligence Magazine - February 2022 - 16
IEEE Computational Intelligence Magazine - February 2022 - 17
IEEE Computational Intelligence Magazine - February 2022 - 18
IEEE Computational Intelligence Magazine - February 2022 - 19
IEEE Computational Intelligence Magazine - February 2022 - 20
IEEE Computational Intelligence Magazine - February 2022 - 21
IEEE Computational Intelligence Magazine - February 2022 - 22
IEEE Computational Intelligence Magazine - February 2022 - 23
IEEE Computational Intelligence Magazine - February 2022 - 24
IEEE Computational Intelligence Magazine - February 2022 - 25
IEEE Computational Intelligence Magazine - February 2022 - 26
IEEE Computational Intelligence Magazine - February 2022 - 27
IEEE Computational Intelligence Magazine - February 2022 - 28
IEEE Computational Intelligence Magazine - February 2022 - 29
IEEE Computational Intelligence Magazine - February 2022 - 30
IEEE Computational Intelligence Magazine - February 2022 - 31
IEEE Computational Intelligence Magazine - February 2022 - 32
IEEE Computational Intelligence Magazine - February 2022 - 33
IEEE Computational Intelligence Magazine - February 2022 - 34
IEEE Computational Intelligence Magazine - February 2022 - 35
IEEE Computational Intelligence Magazine - February 2022 - 36
IEEE Computational Intelligence Magazine - February 2022 - 37
IEEE Computational Intelligence Magazine - February 2022 - 38
IEEE Computational Intelligence Magazine - February 2022 - 39
IEEE Computational Intelligence Magazine - February 2022 - 40
IEEE Computational Intelligence Magazine - February 2022 - 41
IEEE Computational Intelligence Magazine - February 2022 - 42
IEEE Computational Intelligence Magazine - February 2022 - 43
IEEE Computational Intelligence Magazine - February 2022 - 44
IEEE Computational Intelligence Magazine - February 2022 - 45
IEEE Computational Intelligence Magazine - February 2022 - 46
IEEE Computational Intelligence Magazine - February 2022 - 47
IEEE Computational Intelligence Magazine - February 2022 - 48
IEEE Computational Intelligence Magazine - February 2022 - 49
IEEE Computational Intelligence Magazine - February 2022 - 50
IEEE Computational Intelligence Magazine - February 2022 - 51
IEEE Computational Intelligence Magazine - February 2022 - 52
IEEE Computational Intelligence Magazine - February 2022 - 53
IEEE Computational Intelligence Magazine - February 2022 - 54
IEEE Computational Intelligence Magazine - February 2022 - 55
IEEE Computational Intelligence Magazine - February 2022 - 56
IEEE Computational Intelligence Magazine - February 2022 - 57
IEEE Computational Intelligence Magazine - February 2022 - 58
IEEE Computational Intelligence Magazine - February 2022 - 59
IEEE Computational Intelligence Magazine - February 2022 - 60
IEEE Computational Intelligence Magazine - February 2022 - 61
IEEE Computational Intelligence Magazine - February 2022 - 62
IEEE Computational Intelligence Magazine - February 2022 - 63
IEEE Computational Intelligence Magazine - February 2022 - 64
IEEE Computational Intelligence Magazine - February 2022 - 65
IEEE Computational Intelligence Magazine - February 2022 - 66
IEEE Computational Intelligence Magazine - February 2022 - 67
IEEE Computational Intelligence Magazine - February 2022 - 68
IEEE Computational Intelligence Magazine - February 2022 - 69
IEEE Computational Intelligence Magazine - February 2022 - 70
IEEE Computational Intelligence Magazine - February 2022 - 71
IEEE Computational Intelligence Magazine - February 2022 - 72
IEEE Computational Intelligence Magazine - February 2022 - 73
IEEE Computational Intelligence Magazine - February 2022 - 74
IEEE Computational Intelligence Magazine - February 2022 - 75
IEEE Computational Intelligence Magazine - February 2022 - 76
IEEE Computational Intelligence Magazine - February 2022 - 77
IEEE Computational Intelligence Magazine - February 2022 - 78
IEEE Computational Intelligence Magazine - February 2022 - 79
IEEE Computational Intelligence Magazine - February 2022 - 80
IEEE Computational Intelligence Magazine - February 2022 - 81
IEEE Computational Intelligence Magazine - February 2022 - 82
IEEE Computational Intelligence Magazine - February 2022 - 83
IEEE Computational Intelligence Magazine - February 2022 - 84
IEEE Computational Intelligence Magazine - February 2022 - 85
IEEE Computational Intelligence Magazine - February 2022 - 86
IEEE Computational Intelligence Magazine - February 2022 - 87
IEEE Computational Intelligence Magazine - February 2022 - 88
IEEE Computational Intelligence Magazine - February 2022 - 89
IEEE Computational Intelligence Magazine - February 2022 - 90
IEEE Computational Intelligence Magazine - February 2022 - 91
IEEE Computational Intelligence Magazine - February 2022 - 92
IEEE Computational Intelligence Magazine - February 2022 - 93
IEEE Computational Intelligence Magazine - February 2022 - 94
IEEE Computational Intelligence Magazine - February 2022 - 95
IEEE Computational Intelligence Magazine - February 2022 - 96
IEEE Computational Intelligence Magazine - February 2022 - 97
IEEE Computational Intelligence Magazine - February 2022 - 98
IEEE Computational Intelligence Magazine - February 2022 - 99
IEEE Computational Intelligence Magazine - February 2022 - 100
IEEE Computational Intelligence Magazine - February 2022 - 101
IEEE Computational Intelligence Magazine - February 2022 - 102
IEEE Computational Intelligence Magazine - February 2022 - 103
IEEE Computational Intelligence Magazine - February 2022 - 104
IEEE Computational Intelligence Magazine - February 2022 - 105
IEEE Computational Intelligence Magazine - February 2022 - 106
IEEE Computational Intelligence Magazine - February 2022 - 107
IEEE Computational Intelligence Magazine - February 2022 - 108
IEEE Computational Intelligence Magazine - February 2022 - 109
IEEE Computational Intelligence Magazine - February 2022 - 110
IEEE Computational Intelligence Magazine - February 2022 - 111
IEEE Computational Intelligence Magazine - February 2022 - 112
IEEE Computational Intelligence Magazine - February 2022 - 113
IEEE Computational Intelligence Magazine - February 2022 - 114
IEEE Computational Intelligence Magazine - February 2022 - 115
IEEE Computational Intelligence Magazine - February 2022 - 116
IEEE Computational Intelligence Magazine - February 2022 - Cover3
IEEE Computational Intelligence Magazine - February 2022 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com