IEEE Computational Intelligence Magazine - February 2022 - 65

Efficiency U E i =1
= >
Equality E=1
N
/Ri
T
H
(5)
Predator
// RR
i 11
-
==
2NRi
i =1
N
/
Sustainability S EE1 tt tr 0
i
= = / Gi ,where =
N
N 1
i =
Predator 2
Finally, additional experiments were executed on different
Harvest checkpoints to explore how agents cooperate. Each
experiment will be detailed in the subsections below. More
details on the experiments can be found in the Supplementary
Materials.
B. Experiment 1: Agents with Identical
Settings in Multiagent Particle
1) Environment Settings
The goal of this experiment is to answer RQ1 and RQ2
(Section I), i.e., whether the Shapley values of predator
agents correlate with the number of times they catch the
prey, and whether these are a close approximation to the
exact Shapley values.
First, when training a model of three predators using default
settings on the Predator-Prey scenario, one could think that
each predator should provide a similar contribution, as predators
do not have significant differences on their speed, action
space or training method. However, in contrast to this assumption,
statistics in Figure 1 show that the performance of each
predator agent (i.e., the number of times each predator catches
the prey) varies significantly: Predator 3 has a larger contribution
than Predator 1, which has a higher contribution than
Predator 0. In fact, the trained MADDPG model developed a
strategy on which Predator 3 and Predator 2 perform better
than Predator 0 at catching the prey. This can be explained by
the fact that MADDPG provides the same reward to all agents,
instead of only rewarding the highest-contributing agent. This
can bias the training, as explained in [24]. On the contrary,
rewarding only the agent who contributed the most does not
highlight or recognize team strategies where the contribution
of every agent was critical to the shared payout.
2) Shapley Values Analysis
Figure 2 shows Shapley values computed for each agent on
each of the five models, over 1,000 sample episodes per model.
This can be observed in a more convenient way in Figure 1,
which shows Shapley values computed for each agent on a single
model. As hypothesized, the decreasing order of agents'
Shapley values is the following: Predator 2, Predator 1 and
Predator 0. Thus, this first experiment supports RQ1, since
Shapley values are able to correctly map contributions to
agents' usefulness in a cooperative multi-agent setting.
1,000 2,000 3,000 4,000
Number of Times the Prey Get Caught
(a)
Predator 0
[ |] (7)
t
i
2
N N ij
j
(6)
Predator 1
Predator 1
Predator 2
02 46 81012
Shapley Value (Contribution of Each Agent)
(b)
FIGURE 1 Predator-Prey environment: Agents are endowed with the
same speed. Left plot shows predator agents mean performance comparison
out of 10,000 sample episodes over 5 different models (run
2,000 sample episodes each) while right plot presents the Monte
Carlo estimation of Shapley values obtained for each predator agent
(M = 1,000; " random " player exclusion method; mean over 5 models).
Predator 0
20
15
10
5
-5
Method
Noop (MC Estimation)
Random (MC Estimation)
Replace (MC Estimation)
Noop (Real)
Random (Real)
Replace (Real)
FIGURE 2 Predator-Prey environment, same agents' speeds. Comparison
of MC approximation of Shapley values with the exact Shapley
values. Each point relates to one of the 5 different models run.
Predator 1
Predator 2
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 65
Agent Names
Agent Names
Shapley Value
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
