Computational Intelligence - February 2017 - 50

effective and cooperative multi-agent
systems for the environments requiring
both parallel and distributed control.
Bagnall et al. [97] used XCS to
model game-playing agents to study the
bidding policies in an electricity market.
Different XCS-based agents were introduced based on the type of electricity
production (e.g., nuclear and gas). The
agents interacted in a simulation setting
with the goal of learning an optimal
bidding strategy, across different electricity production scenarios, to optimize
two different objectives: to minimize
capital losses and to maximize daily
profit. The agent architecture used two
separate populations, each aiming to
approximate the two objective functions
independently. An agent controller was
used to determine the final action (i.e.,
the per unit bidding price), based on
the prediction obtained from the
underlying populations. The experiment's results showed that the agents
were able to adapt to different marketing mechanisms. Some evidence of
evolving a cooperative behavior was
also found under certain scenarios.
Interest in the role of agent-based systems in an electricity market has been
renewed since the popularization of the
smart-grid concept [98]. Bagnall's work
demonstrated the potential of gametheoretic LCS agent models for this
important domain.
Meng and Pakath [99] and later,
Gaines and Pakath [100], [101] used
CS-1 and XCS, respectively, to learn
strategies in the traditional IPD game.
The experiment's results showed better
perfor mance of LCS-based agents
against pre-scripted strategies, including
the infamous Tit-for-Tat. A comparison
of the two LCS approaches (i.e., CS-1
and XCS) was performed in a study by
[101]. The results showed that XCS performed better than CS-1 when dealing
with deterministic opponents, while
CS-1 performed better when dealing
with stochastic opponents. IPD is one
of the most studied games in EGT
[102], with evolutionary algorithms
commonly used to evolve evolutionarily stable strategies. However, most
work in this area has focused on the

monolithic agent models, with a single
population used to evolve the best strategy. LCS, as shown in the above studies,
provide an intuitive and clear way to
extend this work in a multi-agent setting, allowing the investigation of heterogeneous agents' interactions in the
iterative game environment.
Hercog and Fogarty [29] used a ZCS
in a multi-agent simulation setting to
study emergent dynamics in an "El Farol
Bar" (EFB) problem. EFB, later generalized as "minority games" [103], models
social dilemma situations in which
unique game solutions may not exist
and agents may need to take a more
practical approach to reach a decision,
such as forming hypotheses or learning
from experience. This leads to a continuously evolving equilibrium. More
recently, Hercog [104] extended the
above study and used an XCS to model
agents for an EFB variant that considered
multiple bars. The authors contended
that such modeling and simulation studies could be beneficial for benchmarking
multi-agent learning and analyzing
coordination problems. This work highlighted the ability of game-theoretic
LCS agents to provide insights into both
micro- and macro-level behaviors in
complex dynamics.
Takadama et al. [105] used a Pittsburgh-style LCS for modeling an evolutionary agent in a "bargaining game."
Although the focus of this study was
the validation of multi-agent simulations, it was one of the few pieces of
work reported in this review to use
Pittsburgh-style LCS in a game setting.
Three different agent learning architectures were tested, including LCS, evolutionary strategy and an RL-based agent.
The bargaining game was used as the
simulation model. This work highlighted that because of high variation
between multiple simulation runs, more
research is needed to design LCS methods that can be validated in these simulation environments.
Mailliard et al. [106] used a simplified version of CS-1 as a learning
mechanism for social behavioral rules
in an organized theory of actions
framework. Their simplified LCS

IEEE ComputatIonal IntEllIgEnCE magazInE | FEbruary 2017

implemented neither a GA for rule
discovery nor the bucket-brigade algorithm for the credit apportionment.
Instead, they focused on modeling the
action selection in LCS, based on different social satisfaction levels, computed as a function of stakes or
weights associated with each social
relationship and the corresponding
expected payoffs. An exploratory study
was conducted using the standard PD
game to investigate the types of action
strategies learned by LCS and the corresponding effects on the game
dynamics. This work showed the utility
of LCS in performing strategic analysis
using game-theoretic models.
Xianyu and Yang [107] used LCSbased agents in a co-evolutionary spatial
game-theoretic setting to study the fairness behavior in an "ultimatum game."
In contrast with classical EGT, which
allows homogeneous mixing of agents,
spatial games [108] provide a useful platform to model and study the effect of
neighborhood structure on the evolution of cooperation. The conventional
approaches use replicator dynamics to
model evolutionary dynamics in spatial
games. This work was one of the few
studies to explore the use of machine
learning agents, and perhaps the only
one after Seredyski et al. [58] to explore
the use of LCS in spatial games. CS-1
was used to model agents and the game
was studied on both small-world and
scale-free networks under perfect and
imperfect information conditions. This
work further supported the observations
we made above on Pakath's work [99],
that LCS provide far richer modeling
opportunities for EGT than conventional GA-based models. In summary,
the selected work reviewed in this section has highlighted the strengths of
LCS as a versatile modeling framework,
flexible knowledge representation,
learning mechanism and communication interface for studying complex
dynamics using strategy games.
VI. Discussion

Table 1 summarizes the literature survey
presented in Section V across three features: type and sub-type of games to

Table of Contents for the Digital Edition of Computational Intelligence - February 2017