IEEE Computational Intelligence Magazine - November 2021 - 62

V. Experimental Study
To evaluate the effectiveness and efficiency of the proposed
memetic multi-agent framework with human-like environmental
attention control and bidirectional imitation, experiments
are conducted on a commonly used minefield
navigation platform as well as a commercial first-person shooter
game called Unreal Tournament 2004.
A. Minefield Navigation Task
Platform. A Minefield Navigation Task (MNT), depicted in
Figure 5, is selected as the experimental platform. The environment
contains agents, mines, and a target, which are all randomly
generated at the beginning of each learning trial. The
goal of agents is to navigate through a complex environment to
a target location in a given number of time steps. Notably, at
the beginning of the learning process, the agents have no
knowledge of the task and learn along with the learning trials
undertaken. Each agent is equipped with sonar sensors that can
capture environmental information such as a target direction,
mines, and other agents. The agent's sensors can sense a 180°
forward view, including left, left oblique, front, right oblique,
and right. In addition, all agents have the same action space
A = {left, left front, front, right front, right}.
Reinforcement Learning Machine. A self-organizing neural
network, namely FALCON, is employed as the internal learning
machine of memetic agents. In the network, agents use the
vigilance FALCON hyper-parameters t = (. ,. ,. ),
k ,,= 12 3 as their enhanced decision machine. In particular,
k 02 02 05
FALCON's vigilance parameters control the knowledge generalization
ability, resulting in significantly different learning performance.
In general, increasing the vigilance value can
improve the learning accuracy, but at the cost of creating more
memorial neurons.
Configurations. When an agent successfully reaches a target,
it receives an immediate reward value of 1. If the agent
hits a mine, conflicts with another agent, or fails to reach the
goal within the specified time, it receives an immediate
reward of 0. For each experiment setting, 30 groups of independent
experiments were conducted over a total of 2000
learning missions.
The parameter settings of FALCON learning agents configured
in the present experimental study are summarized in
Table I. For fairness of comparison, the parameter settings of
the MNT problem and the FALCON learning model were
configured to be consistent with the previous studies of
MeMAS and FALCON [11], [25], [26]. The results related to
the following metrics will be reported.
❏ SR: The average success rate of an agent in completing the
MNT task.
❏ MN: The average number of memotypes in the agent's mind
universe. For agents with similar SR, the fewer memotypes
they have, the better their knowledge generalization ability.
❏ IN: The average number of social interactions (i.e., imitations)
in the learning process.
Reach Target
(Success)
A
Hit Mine
A
Collision
A Agent
FIGURE 5 Generic illustration of the MNT platform.
TABLE I Parameter Settings of FALCON Learning Agents.
FALCON PARAMETERS
CHOICE PARAMETERS (, ,)12 3
bbb
aa a
LEARNING RATES ( ,,)
123
CONTRIBUTION PARAMETERS (, ,)12 3
VIGILANCE PARAMETERS (, ,)12 3
cc c
tt t
LEARNING PARAMETERS
LEARNING RATE a
DISCOUNT FACTOR c
ACTION POLICY PARAMETERS
INITIAL e VALUE
e DECAY RATE
(. ,. ,. )
10 10 10
01 01 01
(. ,. ,. )
(. ,. ,)05 05 0
(. ,. ,. )
02 02 05
0.5
0.1
0.5
0.0005
❏ SN: The ratio of increased SR to increased MN compared
to the Baseline MAS method:
SN (D) =
MN MN Baseline
SR SR Baseline
() ()
() ()
D
D
-
-
(17)
A higher SN is preferred as it means better generalization of
memotype knowledge.
❏ SI: The ratio of increased SR to IN, indicating the efficiency
of imitation for improving SR:
SI (D) =
SR SR Baseline
IN()
() ()
D
D -
(18)
❏ RT: The average running time for completing 2000 learning
trials of the MNT task. All approaches were run on a
PC with Intel(R) Core(TM) i7-9700K CPU @ 3.60 GHz
and 16 GB RAM.
1) Results of Attention Intensity Control
The first set of experiments aimed to investigate the efficiency
and effectiveness of the proposed attention intensity control
method in MeMAS based on an adaptive intensity attenuation
parameter d (labeled as
d-MeMA ).S Specifically, a simple scenario
was considered in which a single agent learns with
immediate reward feedback in a 16 × 16 minefield. Note that
in this scenario, the sonar sensors of agents comprise two major
components for detecting mine and target information, respectively.
Hence, {, }.
dd targetd
=
mine
62 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2021
IEEE Computational Intelligence Magazine - November 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2021
