IEEE Computational Intelligence Magazine - February 2023 - 89

where NFEs is the number of fitness
evaluations used by the algorithm, Gmax
is the number of generation conducted
in NFEs, f~fk
bestgGmax
k¼0
where N is the initial population size,
and bxc means the greatest integer less
than or equal to x. For t ¼f1; 2; 3; 4g,
is the union set of
the best solutions found in each generation,
and Gt is the algorithmic parameter
including the means of F and CR, H
and p. The form of LSHADEf is the
same as LSHADEo.
Further the LSHADE with HPSS is
named as LSHADEc, which is the combination
of LSHADEf and LSHADEo
and it has the similar form:
½Xt; f~fk
bestgGmax
k¼0 ; Gt¼
LSHADEcðXt1;f; NFEs; Gt1; nfessÞ;
where nfess
is the number of fitness
evaluations assigned for LSHADEf.
By applying the proposed framework,
the resultant algorithm is named
as Q-LSHADE and summarized in
Algorithm 2. In Algorithm 2, LSHADE
with fixed population size (i.e.,
LSHADEf) is carried out for 0:2
maxNFEs (line 5). The best function
valuefbest is updated afterwards (line 6).
The state is then computed according to
the obtained best solutions found in
each generation (line 7). The agent,
which is a Q-table, is used to output an
action (line 11) to decide whether to use
LPSR or not. In case at ¼ 1, LSHADEo
is run for ð1 0:2tÞ maxNFEs. Otherwise,
LSHADEf is run for another 0:2
maxNFEs. Here T is the horizon limit
which is set to be 4.
C. Training the Agent
The evolution process of Q-LSHADE
is modeled as an MDP and the Q-learning
algorithm is used to train the agent.
In the following, the MDP components
for Q-LSHADE are presented, including
state, action, transition probability
and reward. fk
best represents the minimum
function value obtained up to the
k-th generation.
State: st is a concatenation of s1
s2
t , i.e., st ¼½s1
t ; s2
follows. Denote
gt ¼
0:2 t maxNFEs
N
t and
t, which are defined as
þ 1
s1
t ¼
¼
where s1
t
log fgt50
best
log f0
log fgt
log fgt50
best
log f0
best
best log fgt
best
is used to measure the difference
between the best function values
in adjacent 50 generations; s2
t measures
the descent rate from the first population.
Eq (3) realizes the Represent ()
function in line 7 ofAlgorithm 2.
Action: The action space A is
f0; 1g. That is, the agent can either
choose to implement LPSR (at ¼ 1) or
not (at ¼ 0) at the t-th time step.
Transition Probability: Given a
horizon limit T,when t < T 1and
at ¼ 0, the next state stþ1 is defined as
above. However, in case at some t < T,
at ¼ 1, and t ¼ T 1; at ¼ 0, stþ1 will
be the " terminal state " since in both cases
Q-LSHADE will implement the LPSR:
t < T, at ¼ 1 implies Q-LSHADE
Algorithm 2. Pseudo Code of Q-LSHADE
Input: Initial parameters G0, the maximum number of fitness evaluation maxNFEs,
objective function f and the agent Q
Output: fbest
1: Set fbest þ1, a0 0;
2: Initialize population X0 ¼fx0
½Xt; f~fk
bestgGmax
i ; 1 i Ng randomly;
3: Set G LSHADEf; Initialize G0;
4: for t ¼ 1 ! T do
5:
6:
fbest minffbest; fðXt
8:
9:
7: st Representðflog ð~fk
if Qðst; 0Þ¼¼ Qðst; 1Þ then
Take action at randomly;
10: else
11:
12: end
13: if at ¼¼ 1or t ¼¼ T then
14: G LSHADEo;
15:
Exit;
16: else
17: G LSHADEf;
18: end
19: end
20: ½Xf; f~fk
bestgGmax
k¼0 ; Gt GðXt1; f; 0:2 maxNFEs; Gt1Þ;
i Þ; 1 i Ng;
bestÞgGmax
k¼0 Þ;
best
; s2
t
(3)
implements LPSR at t-th time step. t ¼
T 1; at ¼ 0 implies that Q-LSHADE
does not implement LPSR at t ¼ T 1.
However, thetimehorizon is setas T,
which implies Q-LSHADE implements
LPSR at t ¼ T. Thus t ¼ T 1; at ¼ 0
impliesQ-LSHADEusesLPSRat t ¼ T.
Reward: The agent will have no
effect on the search after terminal state.
If stþ1 is a terminal state, the following
cases are considered.
❏ If t ¼ T 1 and at ¼ 0, the reward
rtþ1 will be the negative logarithm
of the minimum function value
found by LSHADEc
with nfess ¼
0:2 T maxNFEs.
❏ If t < T and at ¼ 1, reward rtþ1 will
be the negative logarithm ofthe minimum
function value found by
LSHADEc
with
nfess ¼ 0:2 t
maxNFEs.
If stþ1 is not a terminal state, the
reward is set zero since the algorithm's
performance is not observed before terminal
state.
The training process is summarized
in Algorithm 3. In the algorithm, to
handle the scale problem (i.e., different
at arg maxa2f0;1gfQðst; aÞg;
k¼0 ; Gtþ1 GðXt; f; ð1 0:2tÞmaxNFEs; GtÞ;
21: return fbest ¼ minffbest; ffðxiÞ;xi 2 Xfgg.
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 89
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
