IEEE Computational Intelligence Magazine - February 2023 - 91

Algorithm 4. The Pseudo-Code of DQ-HSES
Input: an optimization function fðxÞ; x 2 R
1: Set Idx 2 R
Output: an optimal solution x
ed ~0; t 0; a0 0;
2: while at 6¼ 1 and t < T do
3: Xtþ1 UniSamplingðXt; f; Idx; 10Þ;
4:
5:
6:
t t þ 1;
st RepresentðXt; log ðfðXtÞÞÞ;
at arg maxaDQNðst; a; wÞ;
7: end
8: ½Xt2
10: ½Xt3
; x
9: Idx DetectðXt2
; x
11: return x x
2 CMA-ESðXt; uÞ;
Þ;
3 UniSamplingðXt2
3.
; f; IdxÞ;
Action: The action spaceA is f0; 1g.
ed, initial population X0 2 R
maximum number of evaluations maxNFEs
edN, and the
That is, the agent can either choose to
switch (the action takes 1) to CMA-ES or
not switch (the action takes 0).
Transition Probability: In case at
at ¼ 1, and t ¼
some t < T,
T 1; at ¼ 0, stþ1 will be the " terminal
state " . The horizon T constrains the
maximal computational resources used
for the first univariate sampling.
Reward: Three cases are considered
B. DQ-HSES
Here we present how to use a Deep Q
Network (DQN) to control the switching
time in HSES. Algorithm 4 summarizes
the pseudo-code of the developed
algorithm (named as DQ-HSES).
In DQ-HSES, whether to switch
from univariate sampling to CMA-ES is
judged in every 10 generations. In
Algorithm 4, univariate sampling is first
carried out for 10 generations (line 3).
The obtained function values are summarized
to obtain the current state
(line 5). The DQN with the learned
parameter w determines an action in
line 6, which takes a value of 0 or 1. If
the action is 1, the switch happens; otherwise
the univariate sampling is carried
out again. The rest of the components,
including the parameter settings, are the
same as in HSES. Notice that in the
original HSES, the fitness evaluation
number of CMA-ES is limited to
maxNFEs/2. In our experiment, we
guarantee the evaluations for the first
univariate sampling is not larger than
maxNFEs/2 by setting T for the first
univariate sampling procedure. The
flow chart of DQ-HSES is shown in
Figure 3, in which the RL agent is used
to control when to escape from the univariate
sampling phase.
C. Training the SwitcherAgent
To find the optimal weights (denoted as
w) for the DQN, which is called the
switcher agent, deep Q-learning is used
by modeling the evolution process of
DQ-HSES as a finite-horizon MDP
with continuous state and discrete
action space. fk
best represents the minimum
function value obtained up to the
k-th generation.
State: st is a concatenation of s1
t and
s2
t which are defined as follows. When
t > 1,
s1
t ,
s2
t ,
log f10ðt2Þ
best
log f0
log f10ðt2Þ
log f10t
best
best
log f0
1 ,
t
best
best log f10t
best
In case t ¼ 1, s1
the definition, s1
log ðf0
bestÞlog ðf10
jlog ðf0
is used to measure the
bestÞj
difference between the best function
values in adjacent 20 generations; s2
t
measures the descent rate from the first
population.
The state st is thenmodified as a combination
of various bins including ½10s1
10s2
t; ½10s1
t ; 10s2
t; ½s1
t ; 200s2
t; ½s1
t ; 200s2
t
for 10D, 30D, 50D and 100D problems,
respectively.
t ;
bestÞ
: In
;
(4)
when defining the reward. For the terminal
state stþ1,
❏ if t ¼ T 1; at ¼ 0 the reward rtþ1
will be the negative logarithm ofthe
minimum function value found by
HSES by switching at the 10Tth
generation.
❏ if t < T and at ¼ 1, the reward rtþ1
will be the negative logarithm ofthe
minimum function value found by
HSES by switching at the ð10tÞ-th
generation.
For non-terminal state, its reward is
set to zero.
1) The Training Details
Given the above tuple definitions, the
DQL training procedure can be summarized
in Algorithm 5. Algorithm 5 differs
from the classical DQL in two aspects.
First, in classical DQL, the network takes a
state as input. Its output is a vector which
has the same dimension as the action space.
Each component ofthe output vector corresponds
to the Q-value for each action
[50]. This means that the weights in the
DQN except the last layer are shared for
all actions. The shared weights can extract
some common features of the state, but
such structure could sacrifice the DQN's
learning capacity. To address this problem,
two networks Q1ðS; wÞ and Q2ðS; wÞ
are used to replace QðS; 0; wÞ and
QðS; 1; wÞ, respectively.
Second, a DQN is trained for each
training function rather than a DQN for
all the functions. This is to eliminate the
interferences among different learned
DQNs due to the diverse ranges ofdifferent
training functions.
Notice that in line 3, the trajectory
m is generated similarly to that in the
Trl
FIGURE 3 The flow chart of the proposed approach DQ-HSES.
training process for Q-LSHADE (Algorithm
3). The difference is implementing
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 91
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
