IEEE Computational Intelligence Magazine - February 2023 - 90

functions have different ranges of values),
Qðs; aÞ is used to aggregate the Qtables
qðs; aÞ trained for each function.
In Algorithm 3, for each training
function fl , T trajectories Trl
m; m ¼
1; 2; .. . ; T are first generated (line 4 to
Algorithm 3. The Training Process for Q-LSHADE
Input: Training functions f1; .. . ; fL, the maximal number of epochs maxE, the horizon
T and the learning rate a
Output: Q-table Qðs; aÞ
1: Initialize Qðs; aÞ 0 for all s 2S; a 2f0; 1g;
2: for l ¼ 1 ! L do
3:
4: for m ¼ 1 ! T do
5:
6:
7:
bestgGmax
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
end
fk
ffk
best 1 P51
calculate fstgm
51
bestgGmax
frtgm
Trl
else
frtgm1
Trl
end
19: end
20: // Update the Q table qðs; aÞ
21: for e ¼ 1 ! maxE do
22:
23:
24:
for t ¼ 1 ! T 1 do
if t ¼¼ T 1 then
qðst; 0Þ ð1 aÞqðst; 0Þþ aðrtþ1Þ where
fstg2 Trl
T, rtþ1 2 Trl
25:
26:
27:
28:
29:
else
qðst; 0Þ
ð1 aÞqðst; 0Þþ amaxatþ1
where fstg2 Trl
end
qðst; 1Þ ð1 aÞqðst; 1Þþ aðrtþ1Þ where
fstg2 Trl
T, rtþ1 2 Trl
end
30: end
31: // Aggregate the Q-tables for the training functions
32: for s 2S do
33:
34:
35:
36:
37:
38:
if qðs; 1Þ > qðs; 0Þ then
Qðs; 1Þ Qðs; 1Þþ 1;
end
if qðs; 1Þ < qðs; 0Þ then
Qðs; 0Þ Qðs; 0Þþ 1;
end
39: end
40: end
41: return Qðs; aÞ.
90 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
t;
T;
qðstþ1; atþ1Þ,
T;
t¼1 0 and rmminðffk
bestgGmax
m fs1; a1; r2; .. . ; sm1; am1; rmg;
k¼0 Þ;
k¼0 ;
set at 0; t < m and am 1;
if m < T then
t¼1 0 and rmþ1minðffk
m fs1; a1; r2; .. . ; sm; am; rmþ1g;
bestgGmax
k¼0 Þ;
for i ¼ 1 ! 51 do
Randomly initialize X0;
f~fi;k
k¼0
LSHADEcðX0; fl; maxNFEs; G; 0:2m maxNFEsÞ;
i¼1 log ð~fi;k
bestÞ; k ¼ 0; .. . ; Gmax;
t¼1 according to Eq. (3) by using
Initialize qðs; aÞ 0 for all s 2S; a 2f0; 1g;
//Create trajectories for each training function:
line 17). Each trajectory Trl
as follows: first LSHADEc
m is generated
is implemented
with nfess ¼ 0:2m maxNFEs
for 51 times. The best function
values in each generation f~fi;k
bestg51
i¼1; k ¼
0; ... ; Gmax (line 7) are recorded.
The mean of the logarithm
of these best function values, i.e.,
1
51
P51
i¼1 log ð~fi;k
bestÞ; k ¼ 0; .. . ; Gmax
is
computed (line 9). The state, action and
reward are then obtained by summarizing
these values according to the
equations defined beforehand. With
these trajectories, the Q-table qðs; aÞ
is updated from line 24 to line 28.
Finally Qðs; aÞ is updated by aggregating
the trained Q-tables qðs; aÞ
(line 34 and line 37). After training,
Qðs; aÞ is used for Algorithm 2.
Remark. Notice that from
line 13 to line 17, the recorded trajectories
are divided into two cases
(m < T and m ¼ T). The two cases
are corresponding to the two cases of
reward. For m < T,it records the
trajectory generated by LSHADEc
using LPSR at 0:2m maxNFEs. For
m ¼ T, it records the trajectory generated
by LSHADEc
using LPSR at
0:2T maxNFEs. Thus for m ¼ T,the
trajectory only records actions until
aT1,since
aT1 ¼0implies
LSHADEc using LPSR at 0:2T
maxNFEs, which can result in a terminal
state.
V. CONTROLLING THE SWITCHING
TIME IN HSES
In this section, the proposed framework
is applied to control the switching time
in HSES.
A. HSES
HSES [17] is the winner algorithm in the
CEC 2018 competition, in which univariate
sampling and CMA-ES are applied
sequentially. The pseudo-code of HSES
can be found in Supplementary Materials.In
HSES, the timing to switch from univariate
sampling to CMA-ES is very important to
the performance of HSES. In HSES, the
CMA-ES is implemented after using univariate
sampling 100 generations. In the following
discussion, how to control the
switching time by applying the proposed
framework is presented.
http://www.Materials.In
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
