Computational Intelligence - February 2017 - 49

scr ipted agents. The exper iments
showed that XCS was able to learn better strategies than the other agents
could. The authors listed model interpretability, effective generalization, and
the flexibility to allow application to
both the Markovian and non-Markovian environments as key reasons to
choose LCS over other RL methods for
this industrial problem.
Knittel and Bossomaier [88], [89]
introduced LCS called the Activation
Reinforcement Classifier System (ARCS)
and evaluated its capabilities in the
game of Dots and Boxes. ARCS was
based on the concept of reusable features and a modular design to improve
LCS scalability in combinatorial games.
The main idea was to learn small fragments or features from the environment
as rule constructs and then use networks of these features to define
matching rules for given states of the
game. The performance of ARCS and
XCS was compared with increasing
board sizes in the game. Both systems
showed similar learning trends as the
game size was varied. However, ARCS
performed slightly better than XCS in
catching-up speed when the problem
size was changed. Once again, this work
exemplified the flexibility of LCS to be
abstracted to other evolutionary RL
algorithms for meeting specific problem requirements.
LCS applications in combinatorial
games are surprisingly limited, despite
the fact that such games can replace
the traditional synthetic problems used
in basic LCS research in RL problems
easily. In addition to being good
benchmark problems, such games can
become a source of innovation and
improvement in LCS research, as evidenced by the works above.
C. Simulation Games

Simulation games are not common,
owing to their generally serious applications. It is encouraging to see a small
number of LCS applications being used
in this category of games. Kobayashi and
Terano [90] explored the use of XCS in
designing a simulator for business education, as well as an artificial agent that

engaged with the simulator along with
the human users. The simulation results
suggested that XCS was able to learn
better business decisions, which could
lead to better profits, than the other
types of agents (including humans). The
authors also conducted a sensitivity
analysis of XCS parameters and suggested that a higher GA frequency (lower
i GA) and a smaller exploration probability provided better and more robust
outcomes. The authors highlighted the
interpretability of the learnt model as a
key advantage in this set-up, which
allowed an easier integration with
human users and other hand-coded
agents. The rules learned by XCS were
also deemed useful for developing new
game scenarios.
Fernando [91] showed the ability of
LCS to learn lexical and syntactic conventions for effective communication
between two agents in a "language
game." A language game models communication between at least two agents,
with the agents trying to develop a language convention in order to understand each other, to achieve a common
goal. The specific language game in this
work used arbitrary concepts and sounds
to represent language conventions. These
were encoded as alphabetic strings and
integers, respectively. Two implementations of XCS were employed in a coevolutionary framework, one acting as a
transmitter (speaker) and the other as a
receiver (listener). Each XCS, in turn,
consisted of two separate classifier populations, one covering the syntactic rules
and the other covering the lexical rules.
The results showed that the XCS-based
agents were able to learn correctly the
syntactic and lexical rules that represented the communicated concepts. This
work demonstrated the application of
LCS in implementing symbolic communication between agents. The authors
found the incremental addition of new
populations of classifiers an especially
useful feature, as it allowed modification
of syntactical conventions without interfering with lexical conventions. They
also noted that the generalization pressure in XCS helped in learning systematic syntactic conventions [91].

Li and Liu [92] used LCS to model
two types of agents (i.e., institutional
investors and regulators) in a game that
modeled regulatory dynamics in an artificial stock market. The two types of
agents adapted their strategies during
the simulation; that is, while the agents
representing small and medium-sized
enterprises engaged in making investment decisions. Unfortunately, the
authors did not describe the details of
the LCS models used in the simulation,
but listed adaptability as the key strength
that encouraged them to adopt LCS for
this problem.
As with the combinatorial games, the
LCS applications under this category
were limited and warrant further investigation. LCS can be used both as an
agent interacting with other agents in
the simulated environment and as a simulation miner with a goal to continuously improve the simulation and make
it more interesting for its users.
D. Game Theory

Several studies have explored the use of
LCS in game-theoretic setting. One of
the first studies in this respect was presented in [93]. Instead of modeling
game-playing agents, the authors used
the concept of evolutionary stable strategy to provide theoretical foundations
for convergence in LCS. Their empirical
results, using a Hawke and Dove game,
showed that the bucket-brigade algorithm leads to an evolutionary stable
population, as long as the population is
kept static; that is, the GA operation
is suspended.
Seredyski et al. [58] were perhaps the
first researchers to propose the use of
LCS as a game-playing agent. They used
CS-1 to encode agents that engaged in
iterative two-by-two games with their
immediate neighbors. The agents were
placed on a ring to enforce the neighborhood structure. Later, this work was
extended to develop a task scheduler for
parallel computing and was reported in
a series of papers, [94]-[96]. This work
may also be considered the first application of LCS to spatial games. The work
demonstrated the strength of game-theoretic modeling using LCS in designing

FEbruary 2017 | IEEE ComputatIonal IntEllIgEnCE magazInE

Table of Contents for the Digital Edition of Computational Intelligence - February 2017