IEEE Computational Intelligence Magazine - November 2021 - 66

need to monitor their health, use medical items, collect powerful
weapons, and gather enough ammunition to improve their
chances of survival and winning.
Pogamut. Pogamut, introduced to simplify the configuration
of UT2004, is a plug-in for the NetBeans Java development
environment. Pogamut communicates with a UT2004
agent through Gamebots 2004 (GB2004). GB2004 is an additional
modification written in UnrealScript (UT2004's scripting
language), which passes game information to the agent and
collects game information from the agent. The architecture of
Pogamut is shown in Figure 15.
Furthermore, Pogamut provides templates for various types
of simple agents. In UT2004, Pogamut provided a competitive
agent, which was adopted in this study as an experimental
benchmark and FALCON agent opponent. For the purpose of
comparison, in the following experiments, opponent agents
used HunterBot robots provided by Pogamut, which have consistent
combat performance.
Configurations. All experiments were conducted in the
Deathmatch scenario of UT2004. For each setting, 20 groups
of independent experiments were conducted.
First, two sets of the same number of game AIs were initialized
in the region of interest. One group of agents used
memetic FALCON as an online learning machine, and the
other group used HunterBot with human-defined policies as
the opponents. Each agent was equipped with a set of sonar
sensors, which could obtain the current state of the agent and
other information. The basic state of an agent includes its
health level, whether it is being damaged or not, whether enemies
are spotted, ammunition level, and variables that indicate
the current action of the agent by means of archiving previous
states and actions. The state vector contains 18 variables encoded
by Boolean values or discrete numbers within [, ]01 . The 18
variables are divided into nine pairs, each of which is the base
state value and its complement with respect to 1.
After capturing the state information, the agent can make a
choice as to which of the four executable actions to perform,
and then it learns the relationship between the state and
actions. The agent's action space includes explore, find items,
engage, and medkit, which are defined as follows:
❏ Explore action: an agent explores the map randomly.
❏ Find items action: an agent finds and picks up the nearest
visible weapons, armor, ammunition, or other items.
❏ Engage action: an agent takes offensive actions to pursue
and destroy opponents.
❏ Medkit action: an agent recovers its health by finding medical
kits.
When an agent performs an action, it receives a reward
from the game environment. The reward system is defined in
Equation 19. When an agent kills an opponent, the reward is 1,
and when an agent is killed by an opponent, the punishable
reward is 0. In addition, when an agent successfully attacks an
opponent, increases health, or collects a new weapon, it receives
an instant bonus of 0.5.
reward = *
1 ,
05
.,
,
kill the enemy
attack enemyhealget weapon
killed by theenemy
.
,,
.
In order to quantitatively evaluate a team's performance in
combat, agent performance can be measured using the Kill
Death (KD) ratio (see Equation 20), which is calculated by the
percentage of opponent kill numbers over agent death numbers.
In general, a high KD is preferred, as it means that the
agent kills many opponents and dies few times.
FIGURE 14 A screen snapshot of Unreal Tournament 2004.
KD =
NNd
1 Nk
N
/
i =1
Nk i
Unreal
Tournament
2004
Server
GameBots
2004
FIGURE 15 Pogamut architecture overview.
Local
Parser
Agent
Work Station
IDE
()
()
i
i
(20)
where N is the number of players on
each team, ()
Nd i
represents the number
of opponents killed in a game by
the ith player of the team, and ()
represents the total number of deaths of
the ith player of the team.
Objective. In UT2004, memetic
agents were trained to fight against the
rule-based HunterBot agents (Hunters).
Hunters are pre-defined in UT2004
66 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2021
.
(19)
IEEE Computational Intelligence Magazine - November 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2021
