IEEE Computational Intelligence Magazine - February 2022 - 66

3) Comparison of Approximated Shapley
Values with Exact Shapley Values
In this subsection, the Monte Carlo approximation of Shapley
values is compared with the exact computation of Shapley values
to verify RQ2. For that, Eq. 2 is used to compute Shapley
values: marginal contributions are estimated as the mean global
reward obtained by the coalition of players over a high number
of episodes-1,000 episodes in this experiment-using the player
exclusion mechanisms in Section IV. Indeed, a large number of
samples is needed for each coalition due to the stochastic
nature of the environment, which leads to high variance in
results. As depicted in Fig. 2, Shapley values estimated by
Monte Carlo sampling are very close to the real Shapley values.
The difference of only 5% on average for all agents and models
therefore supports RQ2. In addition, MC approach is simpler
to implement and with a complexity that does not grow exponentially
with the number of agents in the RL environment.
C. Experiment 2: Introducing Variations in
Agents Speeds in Multiagent Particle
Predator 0
(Slow)
Predator 1
(Medium)
Predator 2
(Fast)
1) Environment Settings
This experiment introduces variations in the agent settings with
the objective of disturbing the actual distribution of contributions
between agents obtained in Experiment 1 and see if, as claimed in
RQ1, this change will be reflected in the computed Shapley Values
(i.e., the aim is to ensure that Shapley values correlate with the
observed contributions). Thus, the speed of each predator will differ
from the default one and so, it can be expected that the faster
agent will catch the prey more often and contribute the most to
the global reward. In this manner, our goal is to obtain a clear hierarchy
between agents that correlates with Shapley Values so they
reflect the agents' observed behavior coherently. The following
speeds are arbitrarily set for each agent: Predator 0 (slow): 0.2,
Predator 1 (medium): 0.8, Predator 2 (fast): 2.0. Statistics in
Figure 3 show the performance of each predator agent in terms of
the number of times each predator catches the prey. As expected,
the faster agent (Predator 3) presents a higher contribution than
Predator 1, which exhibits a higher contribution than Predator 0
(the slowest). More precisely, the ranking in speed is reflected in
the ranking of contributions, which is the ideal setting to test if the
assumption in RQ1 is valid. In addition, the real Shapley values for
these settings were also computed in order to further verify RQ2.
Number of Times the Prey Get Caught
(a)
Shapley Value for Each Agent
Predator 0
(Slow)
Predator 1
(Medium)
Predator 2
(Fast)
02 46 81012
Shapley Value (Contribution of Each Agent)
(b)
FIGURE 3 Predator-Prey environment, variable agents' speeds. The left
plot shows predator agents' performance comparison for 10,000 sample
episodes and 5 different models (2,000 sample episodes for each of the
five trained models) while the right plot presents the Monte Carlo estimation
of Shapley values (M = 1,000) for each predator agent (averaged
over 5 model runs) with the " random " player exclusion option.
14
12
10
8
6
4
2
0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00
Speed of Predator 0
FIGURE 4 Predator-Prey environment, variable agents' speeds. Monte
Carlo approximation of Shapley values (M = 1,000) obtained by a
single predator agent with respect to its speed.
2) Shapley Values Analysis
Figure 5 shows the Shapley values computed for each agent on
each of the five models, over 1,000 sample episodes per model.
This can be observed in a more convenient way in Figure 3,
which shows the Shapley values computed for each agent on a
single model. Following RQ1, the order of agents' Shapley values
is the following: Predator 2, Predator 1, and finally Predator 0.
These Shapley values accurately correlate with the number of
times each of the agents caught the prey (see Figure 3). In
addition, as noticeable in Fig. 4, when making a single agent's
66 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2022
Agent Names
Agent Names
Shapley Value
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
