IEEE Computational Intelligence Magazine - November 2021 - 60

input x .,ci1
The lower the distance, the higher the value of I ,i
and the clearer the memory.
To elaborate the algorithm, first
di 112 f== is
/,
Li ,, ,
initialized (see line 1 in Algorithm 2). For each step in a learning
trial, the most suitable memotype is activated for instructing
agent decision making according to the meme activation process.
Once a learning trial succeeds, all the activated memotypes are
defined as the useful memes and the corresponding memory
intensity I per step is transferred into the memory intensity set
MIS (see line 11 in Algorithm 2). After certain learning trials,
the intensity values
iL,12f=
,
and the median values Ii
fied foriL1 f= ,, .
It is assumed that there is a causal relationship between the
values of memory intensities in all trials and an agent's learning
performance (i.e., probability to succeed). If an agent remembers
certain channels of state information more clearly than the
others, they are expected to have a higher probability to succeed.
For example, II
AB
Median
2
Median means that the agent is
more likely to succeed if they remember the information from
the Ath sensory channel more clearly. Recall that d controls
the intensity of attention for information from the different
sensory channels. If
dd the information from the Ath
AB ,2
sensory channel tends to have a larger impact on calculating
the activation values Tj
of the memotype j, causing more memotypes
to be generated for remembering the information from
the Ath sensory channel more clearly, which in the end
improves the agent's learning performance.
Intuitively, Ii
Median
represents the intensity of the successful
j
II /R 1
Median = Median
=
ii Ij
L
memory on the information from different sensory channels.
Based on the normalization of
Median for
iL the attention intensity id is then updated (see
line 16 in Algorithm 2).
= 12 f ,, , ,
dd )=- d+ () fitness
() Ii
new
i
()
di 12 f=
iL
oldold
Median ii
Note if the fitness (i.e., success rate) of an agent is poor, Ii
can be inaccurate, and ,, ,, is updated slowly.
()
(11)
Median
in MIS {, ,} are sorted for
Median of MISi are identii
III
iii
= 123
f
IV. Bidirectional Imitation in Meme Selection
In the past, many researchers have focused on answering the
question, " (how) can agents benefit from the useful knowledge
of other agents in order to achieve better individual and overall
system performance? " A significant amount of research has
been proposed to exploit domain expertise with varying
amounts of human-provided knowledge. Many knowledge
sharing strategies such as instance transfer [28], model transfer
[29], and advice exchanging [30] have been proposed, which
benefits a wide range of learning and decision-making tasks.
However, most of these strategies assume that a well-trained or
predefined domain expert is provided. Consequently, they are
not feasible in multi-agent learning scenarios where all agents
begin the learning process tabula rasa [31].
In multi-agent learning, each agent can be unique both memetically
and genetically. While any given agent may not be an expert
during learning, they may have useful local knowledge that all others
are not aware of due to their independent learning experiences
(in the form of memotypes). Similar to the human social learning
scheme, these learning agents would likely benefit from knowledge
shared by their partners, thereby improving the effectiveness of system-wide
learning. Current research is beginning to take steps
toward addressing the problem of knowledge sharing. Several studies
have investigated learning from demonstrations [32], action
advising [33], [34], knowledge broadcasting [35], and imitation [36].
Among them, the superiority of the imitation strategy has been
verified in MeMAS and its enhanced versions for increasing the
learning performance of memetic agents by considering the essential
human learning principles, i.e., " imitate-from-elitist " [10] and
" like-attracts-like " [11]. Still, these approaches always assume that an
agent is capable of constantly monitoring the performance of all
others. The imitation process occurs whenever a potential expert is
identified, that is, an individual with better learning fitness or bearing
certain similarities under the given circumstances. This approach
suffers from drawbacks, as constantly paying attention imposes
increasing communication costs and can be simply unrealistic.
In this study, the objective is to develop a more effective
State Judgment
Imitation
The Imitator Policy
Request
Respond
Own Action ai
Imitated Action aj
Action
Memetic Agent i
FIGURE 4 Illustration of the imitation process.
Memetic Agent j
where A is the space of all available actions, and (, )Qs a l is the
normalized Q-value of performing the action a at a state s:
60 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | NOVEMBER 2021
State Judgment
Imitation
The Imitated Policy
knowledge sharing scheme in multi-agent learning. Different
from the existing MeMAS research, a bidirectionally interactive
imitation strategy (shown in Figure 4) is proposed based on an
agent's estimation of the importance and/or uncertainty of
decision making at a given environmental state.
Recall that a memotype inside an agent's mind universe essentially
encodes a mapping of the input state s and the action a,
through the reward measurement
R = (( ,),Qs a
1 -Qs a (, ))
which is updated using Q-learning functions. Accordingly, a
method is proposed to measure the state importance, Imp(s), of the
current state s using a Gini coefficient regarding the Q-value distribution
for all possible actions, which is formulated as follows:
A
Imps
() / ll(, )( (, ))
1
=-#
=
1
a
QsaQsa
(12)
IEEE Computational Intelligence Magazine - November 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2021
