IEEE Computational Intelligence Magazine - February 2023 - 93

Algorithm 6. Deep Q network based HSES (DQ-HSES)
Input: an optimization function fðxÞ; x 2 R
learned DQNs fQ1
1: Set Idx ~0, t 0;
2: while t < T do
3:
4:
5:
6:
7:
8:
9:
½Xtþ1; x
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
end
if Q1
end
else if Q1
end
end
if Vote1 6¼ Vote0 then
at ðVote1 > Vote0Þ;
else
Take at randomly;
end
if at ¼¼ 1 then
Exit;
end
t t þ 1;
26: end
27: ½Xt2
; x
; x
28: Idx DetectðXt2
29: ½Xt3
30: return x x
2 CMA ESðXt; uÞ;
Þ;
3 UniSamplingðXt2
3.
; f; IdxÞ;
Q-LSHADE is larger than that of
LSHADE. Furthermore, the values of
BM and the results ofthe hypothesis test
on the remaining functions, i.e., functions
excluding F13 and F16,are presented
in the brackets in Table II. When
only the functions excluding the training
functions are considered, Q-LSHADE
still performs significantly better than
LSHADE on four functions and worse
on two functions. The value of BM
obtained by Q-LSHADE is still greater
than that of LSHADE. Thus the proposed
learning-to-optimize based framework
can learn a suitable agent to control
the structural parameter.
VII. EXPERIMENTAL RESULTS ON
DQ-HSES
In this section, DQ-HSES is compared
with its counterpart HSES and some
well-knownEAs on the CEC2018 competition
test suite. Furthermore, to verify
the capability ofDQ-HSES for generalization,
it is tested on the CEC 20143 test
suite. Note that the original HSES is the
winner of the CEC 2018 competition.
The authors ofa review [52] comprehensively
studied various DEs, and concluded
that SHADE, cDE, and CoBiDE
with the rand/1/bin and curr-to-pbest/
1/bin operators are the best three methods.
Therefore, cDE, CoBiDE,and
jSO (an advanced version of SHADE)
are chosen as the baselines.
CoBiDE [30] is an adaptive DE that
adapts the scaling factor F and crossover
rate CR as follows: First, the initial values
3The CEC 2014 test suite can be found at https://
github.com/P-N-Suganthan/CEC2014
CR ¼
randcð0:95; 0:1Þ if randu½0; 1 0:5
randcð0:1; 0:1Þ if randu½0; 1 < 0:5
l ðst; w1Þ < Q2
Vote1 Vote1 þ 1;
l ðst; w2Þ then
l ð; w1Þ; Q2
Output: an optimal solution x
1 UniSamplingðXt; f; Idx; 10Þ;
Compute the state st by Eq. (4);
set Vote0 0; Vote1 0;
for l ¼ 1 ! L do
if Judgeðst; Trl
Q1
T; c; TÞ¼¼ 0 then
l ðst; w1Þ 0; Q2
l ðst; w1Þ > Q2
l ðst; w2Þ 0;
l ðst; w2Þ then
Vote0 Vote0 þ 1;
ofF and CR for each individual are sampled
from the following distributions:
F ¼
randcð0:65; 0:1Þ if randu½0; 1 0:5
randcð1; 0:1Þ if randu½0; 1 < 0:5
(5)
l ð; w2ÞgL
evaluations maxNFEs and a threshold c;
ed, initial population X0 2 R
edN, the
Algorithm 7. Judge
Input: s,Trl
l¼1 and the maximum number of
Outut: output
1: dmin þ1;
2: for i ¼ 1 ! T 1 do
3:
Trl
T;
4: end
5: if dmin > c then
6:
7: else
8:
9: end
return output 0.
return output 1.
dmin minfdmin; ks sik2
2g, si 2
T, c, and time horizon T
(6)
During the evolution, the successful
values ofF and CR are inherited, while
their failed values are re-initialized using
Eq. (5) and Eq. (6), respectively.
cDE [40] is also an adaptive DE. Its
values of F and CR are sampled from a
candidate pool composed of Fpool ¼
f0:5; 0:8; 1g and CRpool ¼f0; 0:5; 1g.
The candidate pool contains nine candidate
pairs in total. They are denoted by
O1 ¼f0:5; 0g; ... ; O9 ¼f1; 1g. In the
tth generation, the probability of selection
of each candidate pair is st;k ¼
Nsucc
P
k þc
i Nsucc
i þc , where Nsucc
k
is the number of
individuals with better fitness values
when the pair Ok is used and c > 0isa
constant used to avoid a singularity.
jSO [27] is an advanced version of
LSHADE [12] and iL-SHADE [26],
which are two variants ofSHADE [11].
It took second place in the CEC 2017
competition.
To train the DQNs, six functions
from CEC 20134, F2; F6 F10,and five
functions from CEC 2018: F1; F10; F15;
F18,and F29 are chosen as training
4The CEC 2013 test suite can be found at https://
github.com/P-N-Suganthan/CEC2013
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 93
https://github.com/P-N-Suganthan/CEC2014 https://github.com/P-N-Suganthan/CEC2013 https://github.com/P-N-Suganthan/CEC2014 https://github.com/P-N-Suganthan/CEC2013
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
