IEEE Computational Intelligence Magazine - November 2021 - 56

❏ An interactive imitation strategy is proposed based on agents'
estimation of the importance and/or uncertainty of decision
making in a given environmental state. With the bidirectional
confirmation of imitation, the effectiveness and efficiency
of meme proliferation can be improved significantly.
❏ To verify the efficacy of the proposed method, comprehensive
experiments are conducted in both a minefield navigation
domain as well as a commercial first-person shooter
video game called Unreal Tournament 2004. Experimental
results show that the proposed method has significantly better
learning performance than state-of-the-art approaches.
The rest of this paper is organized as follows. Section II
describes the basic evolutionary concepts of MeMAS, wherein a
popular self-organizing neural network, namely FALCON, is
employed as the online reinforcement learning machine for
memetic agents. Section III presents the methodology and motivation
of the attention intensity control method, and Section IV
describes the bidirectional interactive imitation strategy for more
efficient meme proliferation. Section V recounts the comprehensive
empirical study of the proposed method in the minefield
navigation domain and in Unreal Tournament. Finally, Section VI
concludes this work and discusses limitations and future work.
II. Memetic Multi-Agent System
with FALCON Dynamics
The word meme can be tracked back to Richard Dawkins in
his book The Selfish Gene [22]. Analogous to genes in genetics
serving as the basic units of biological heredity, memes in
memetics serve as the basic units of cultural information in a
sociological context. In the past few years, the study of memetics
in multi-agent systems, namely memetic multi-agent system
(MeMAS), has exhibited significant superiority in achieving a
good level of efficacy for solving decision-making problems in
complex settings, such as in a simulated minefield navigation
task [11] or in 3D interactive video games [23]. As an essential
backbone of our present study, this section provides a brief
description of MeMAS.
Figure 1 provides an illustrative framework of MeMAS. Particularly,
memes represent the building blocks of the agent's mind
universe, which can be divided into memotypes and sociotypes.
Internally, a memotype is described as an idea or knowledge captured
as memory items or generalized abstractions inside an
agent's mind universe. Externally, sociotype memes are behaviors or
actions expressed by agents that can be discovered by other agents.
Meme representation and meme evolution form the two core
aspects of MeMAS. It then undergoes meme expression and meme
assimilation. Meme expression is defined for individuals to express
their stored neuronal memes as behavioral actions, and meme assimilation
captures new memes by translating corresponding behaviors
into knowledge that blends into the individual mind universe. In
other words, meme representation pertains to what the meme is,
whereas meme expression/assimilation activates/updates the meme
during the learning process. Meme evolution, on the other hand, is
central to behavioral learning. Specifically, meme internal evolution
indicates the self (reinforcement) learning process of memetic
agents. Meme external evolution instructs social interaction among
agents mainly via an imitation process for information propagation
across the population. Meme selection pertains to whom one imitates,
and meme variation concerns how to maintain the diversity
tendency of the memes that are transmitted or assimilated.
A. Meme Representation
Meme representation, an important step in memetic computing,
facilitates the identification of an appropriate representation of
a meme (i.e., as memory items or knowledge) inside an agent's
mind universe.
Existing studies have considered a self-organizing neural
Meme External Evolution
Imitation
Meme Variation
network, namely FALCON [24]-[26], as the online learning
model of memetic agents due to its
well established and fast learning performance.
The mind universe of a
memetic FALCON agent is modeled
as a three-channel neural network
architecture (shown in Figure 2) consisting
of three input sensory fields
,,
FF2 and Fc3
nitive field F2
1
cc
1
1
1
(low-level) and a cog(high-level)
through a
bidirectional conditional path. Specifically,
the sensory field Fc
1
1
represents
1
the current states, the motor field Fc2
Meme Assimilation
Meme Expression
⋅
⋅
⋅
Meme Internal Evolution
FIGURE 1 Illustrative framework of Memetic Multi-Agent System [10].
represents the available actions, and the
feedback field Fc
1
values. F2
stores the learned memes/
nodes, each of which encodes relations
of the state input and behavioral
response to incoming patterns from the
three channels. One node in the field
represents a specific meme in the
agent's mind universe.
56 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2021
3 represents the reward
Meme
Assimilation
Imitation
Meme Assimilation
Meme
Variation
Meme
Selection
Meme
Expression
Meme
Selection
Meme
Expression
IEEE Computational Intelligence Magazine - November 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2021
