IEEE Computational Intelligence Magazine - November 2021 - 61

Qs a (, )l = A
Qs a(, )
(13)
/ (, )Qs a
a =1
Intuitively, if the Q-values for all actions in s are similar, it
does not matter which one is chosen, and s is unimportant.
However, if certain actions in s have significantly higher Q-values
than others, s becomes more important as it is necessary to
choose the most appropriate action in s to perform. Typically,
the larger the Imp(s) value, the less important and more uncertain
the current state. To decide whether a state is important or
uncertain, the threshold T is introduced to control the degree
of the state importance. As
Imps
() [, ],01 T 01!! For sim[,
].
plicity,T .05= is set in the present study.
In theory, there are three ways in which the imitation process
occurs, namely forward (student-initiated), backward
(teacher-initiated), and bidirectional (jointly-initiated) imitation.
The three following subsections provide details of the
three imitation strategies.
1) Forward Imitation
Considering that agents learn from scratch and have insufficient
awareness of the state importance, especially at the early learning
stage, it usually takes a large number of learning trials before a
student agent can initiate imitation based on the importance of
the state. Therefore, a new way for a student agent to decide
whether to initiate an imitation process based on its estimation
of the uncertainty of a given state is developed.
Specifically, the given state s is considered to be uncertain
when Imp(s) is larger than the given threshold T. In this case,
there is little difference in return for the actions the agent takes
in this state. When the following calculation equation is satisfied,
the student initiates the imitation process.
A
/ ll 2
a =1
QsaQsa
studentstudent
# 1 -
(, )( (, )) T
(14)
2) Backward Imitation
Different from forward imitation where a student agent initiates
the imitation based on the uncertainty of a given state, in
backward imitation, a teacher agent who has learned effective
strategies for accomplishing a task shall be able to reasonably
judge the importance of the state and subsequently provide
guidance for conducting the imitation process.
When a teacher agent considers a state to be important, the estimated
reward for performing certain actions from the action space
will be obviously higher than the rewards for other actions. Therefore,
the teacher initiates imitation in the hope that the student will
learn by imitating more appropriate actions. This reduces the likelihood
that the student agent will take the wrong action in an
important state, which would lead to significant reductions in future
rewards. The teacher, who is aware of the state importance, can only
initiate imitation when the following condition is satisfied.
A
/ ll 1
a =1
QsaQsa
teacherteacher
# 1 -
(, )( (, )) T
(15)
3) Bidirectional Imitation
Intuitively, an imitation interaction involves both student and
teacher agents. Hence, this process can be initiated not only by
the student agent but also by the teacher agent. Note that forward
imitation is likely to suffer from the inaccurate Q-value
estimation of the student agent, especially at the early learning
stage when students tend to explore the environment with
high randomness. In backward imitation, the teacher agent is
expected to constantly monitor students' states, which
demands a large amount of communication cost and is impossible
in reality.
A bidirectional imitation method is introduced where the
teacher and student agents can jointly decide whether to
initiate an imitation interaction (see Algorithm 3 for a general
illustration). First, the student agent decides whether to
make an imitation request based on the uncertainty of the
given state (see line 6 in Algorithm 3). Upon receiving the
request, the teacher agent then estimates the importance
of the state and decides whether to make a response and
guide the learning process (see line 10 in Algorithm 3).
By doing this, the teacher agent is not required to pay
constant attention to the students' states, and the impact
of the detrimental or unnecessary imitation incurred by the
noisy Q-estimation is reduced significantly. Specifically, bidirectional
imitation occurs only when a given state is considered
to be uncertain by the student agent and important by
the teacher agent, as defined below.
imitation = '
yes
no
,
,
Imps TImp sT
other
studentteacher
()21/
()
.
(16)
Algorithm 3 Bidirectional Imitation.
1 Initialization: N agents; threshold T.
2 while stop conditions are not satisfied do
3
4
5
6
7
8
9
for 6 agent as a student do
Given the current state s.
Calculate Imps
if Imps T
student ()
2 Equation 12.
student () 2 (uncertain) then
10
11
12
13
14
15
16
17
18
19
20 end
Identify a teacher via meme selection.
Pass state s to the teacher agent.
Calculate Imps
if Imps T
teacher
()
teacher () 1 (important) then
Perform meme transmission.
Pass socitype actions to the student.
else
Perform meme internal evolution.
end
else
Perform meme internal evolution.
end
end
2 Equation 12.
NOVEMBER 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 61
IEEE Computational Intelligence Magazine - November 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - November 2021
