IEEE Computational Intelligence Magazine - February 2023 - 92

Algorithm 5. The training process for the DQN
set as 2.4,1.6,16.7,18.2 for 10D, 30D,
50D and 100D problems, respectively.
Input: Training functions f1; .. . ; fL, the maximal number of epochs maxE, the horizon
limit T and the learning rate a
2ÞgL
l ð; w
1Þ; Q2
l ð; w
l ð; w1Þ; Q2
l¼1
1: Randomly Initialize fQ1
2: for l ¼ 1 ! L do
3:
4:
5:
6:
7:
8:
9:
d1
else
10:
11:
12:
13:
14:
15:
16:
17:
t gmaxfQ1
where fstg2 Trl
d1
end
d2
l ðstþ1; w1Þ; Q2
T;
t rtþ1, where rtþ1 2 Trl
w1 w1 arw1kQ1
w2 w2 arw2kQ2
return Q1
l ð; w
2 w2;
1Þ; Q2
t;
//update the parameters of the DQNs
t k2;
t k2;
l ðst; w1Þ d1
l ðst; w2Þ d2
end
end
18: w
19:
20: end
HSES with G1 ¼ 10 m for 51 times and
recording the best function values in
each generation (G1 is the number of
generation assigned to the first univariate
sampling in HSES), i.e., f~fi;k
bestg51
i¼1; k ¼
0; ... ; Gmax, instead of using LSHADEc
in Algorithm 3.
2) DQ-HSES
By embedding the learned DQNs into
HSES, the resultant algorithm is named
as DQN based HSES (dubbed as DQHSES).
DQ-HSES is summarized in
Algorithm 6. In the algorithm, the univariate
sampling is first implemented for
10 generations. The learned DQNs are
used to determine whether switching to
CMA-ES or not (line 4 to 24). The rest
of the algorithm after switching is the
same as HSES. Likewise, the parameter
settings for CMA-ES and univariate
sampling are the same as in HSES.
Line 5-21 show how to use the L
learned DQNs to determine an action
for an optimization function. Here the
concept of 'boosting' [51] is borrowed
from machine learning. The core idea is
to combine several weak agents for a
strong agent. In our study, the L learned
DQNs can be seen as weak agents.
They vote to decide the action.
To implement the boosting mechanism,
Vote0 and Vote1 are used to
respectively record the number of votes
for selecting action 0 and 1 by the
learned DQNs (line 5 to line 15). The
action is selected based on the greatness
of the two values. If they are equal, a
random action is selected.
To avoid over-fitting, the value of
DQNs is set to be zero when a new state
is far from the states ever met in the
training set (line 8). The function
Judgeðst; Trl
T; c; TÞ,summarized
in
Algorithm 7, is used to make this decision.
The Euclidean distance between s
and the states in the training set Trl
T is
firstly computed (line 3). Ifthe minimum
distance is larger than a fixed constant c,
zero returns; otherwise one returns. The
thought behind is that if a new state st is
too far from the states in the training set,
the prediction could be unstable and
implausible since the training data is not
comprehensive. In the experiment, c is
92 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
1 w1, w
l ð; w
2Þ.
l ðstþ1; w2Þg,
Create T trajectories Trl
for e ¼ 1 ! maxE do
l ð; w2ÞgL
m; 1 m T;
for t ¼ 1 ! T 1 do
//Compute the target
if t ¼¼ T 1 then
t rtþ1, where rtþ1 2 Trl
T;
i¼1
Output: DQNs fQ1
VI. EXPERIMENTAL RESULTS ON
Q-LSHADE
The performance ofQ-LSHADE is first
investigated against its counterpart
LSHADE on the CEC 2018 test suite2
as the benchmark. The test suite contains
29 test functions that can be classified
into four categories: unimodal
functions F1 and F3, multi-modal functions
F4 F10, hybrid functions F11
F20, and composition functions F21
F30. In this section, Q-LSHADE is
tested on 10D test functions and its performance
is compared with LSHADE.
F13 and F16 are taken as the training
functions. The training parameters are
a ¼ 0:005 and maxE ¼ 100; 000. The
other parameters of Q-LSHADE and
LSHADE are the same as in the original
reference [12]. maxNFEs is set to
10,000D. Because Q-learning works
only for MDP with a finite and discrete
state space, the space is divided into
½0; 106, ð106; 105, ð105; 103,
ð103; 101, ð101; 1; ð1; þ1Þ for s1,
and [0,0.1], (0.1,0.25], (0.25,0.4],
(0.4,0.6], (0.6,1.5], ð1:5; þ1Þ for s2.
Table II summarizes the statistics
obtained by Q-LSHADE and LSHADE
on the benchmark functions over 51 runs.
All the statistics in Table II and subsequent
tables are computed based on the error
values (i.e., the difference between the
obtained optimum and the known global
optimum). When the error values are
smaller than or equal to 108,they are
assumedto bezero. TheWilcoxonranksum
hypothesis test between LSHADE
and Q-LSHADE at the 5% significance
level is also listed, where y=x implies that
LSHADE performs significantly better/
worse than Q-LSHADE and
means
that there is no significant difference
between the algorithms. Furthermore,
the value ofBM is listed for recording the
number of functions for which the algorithmobtains
the best mean value.
Table II shows that Q-LSHADE significantly
outperforms LSHADE on four
functions and is worse than it on two
functions. The value ofBM obtained by
2 https://github.com/P-N-Suganthan/CEC2018
https://github.com/P-N-Suganthan/CEC2018
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
