IEEE Computational Intelligence Magazine - February 2023 - 86

CR from bimodal Cauchy distributions
with parameters that are fixed in advance
in each generation for each individual if
the individual does not improve fitness;
otherwise, the parameters are inherited
from the last generation.
Most EAs fixsuchstructuralparameters
as the population size during the evolutionary
procedure [13]. Few studies
have investigated the adaptive control of
the population size. According to a review
by Piotrowski [13], research on the adaptive
control of the population size can be
categorized into four classes. The first class
of methods is based on self-adaptation at
theindividuallevel.In[31],eachindividual
is assigned a coefficientthatisupdated
during the evolutionary process. The
coefficients are used to update the population
size. In the second class ofmethods to
adaptively control the population size,
population size is negatively correlated
with the diversity offitnessineachgeneration
[32]. In the third class ofmethods, the
population size is set based on improvements
in fitness in recent generations [33].
That is, the population size increases ifthe
best fitness does not improve in a number
ofgenerations; otherwise decreases. In the
fourth class, the population size is adaptively
reduced, such as the linear population
size reduction (LPSR) strategy
proposed in LSHADE [12]. The LPSR
strategy hasalsobeenusedinseveral
recently developed adaptive DEs, such as
MPEDE [34], iL-SHADE [26], and jSO
[27] Zhu et al. [35] proposed adaptively
TABLE I The notations used in the paper.
NOTATIONS
// Reinforcement learning
S RD
A Rd
m0
r 2 R
T
st
//Q-LSHADE and DQ-HSES
LPSR
nfes
HPSS
LSHADEf
In Alg. 1, the involved functions, including Collect(),
Represent() and theQ-table or network, depend on
specific EA. In the following sections, this framework is
applied to twowell known hybrid EAs, i.e. LSHADE [12]
andHSES [17].
increasing or reducing the population size
based on observations during the evolutionary
procedure.
Structural parameters other than the
population size also often appear in
hybrid EAs [36], [37], [38]. The frequency
of local search, the number of
fitness evaluations assigned for local
search, the fraction ofindividuals used to
perform local search, and the selection of
the recombination operators can be considered
to be structural parameters in a
hybrid EA. In the algorithm that won
the CEC 2018 competition, the hybrid
sampling evolution strategy (HSES)
[17], the switching time from adaptive
univariate sampling to CMA-ES is also a
structural parameter.
The structural parameters in hybrid
EAs are largely responsible for allocating
computational resources to different
phases of the EA to balance exploration
and exploitation. The adaptive setting of
the structural parameters in hybrid EAs is
often handled manually. Taking EP/LS
[37] as an example, the EP phase terminates
when the value ofthe penalty function
is smaller than a pre-fixed value and
the value ofthe optimization function in
EXPLANATION
D-dimensional state space
d-dimensional action space
initial distribution of the state
the reward
the time horizon limit
the state at the t-th time step
linear population size reduction
the number of fitness evaluations
the hybrid population size strategy
the LSHADE with fixed population size
86 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
NOTATIONS
at
pðatjst; uÞ
pðstþ1jat; stÞ
vðsÞ
Qðs; aÞ
Qðs; a; wÞ
Nmin; Nini
nfess
LSHADEo
LSHADEc
the current generation is smaller than
that in the previous generation. In PSO/
DE [39], the computational resources are
allocated according to the number of
non-dominated solutions obtained. The
univariate sampling phase in HSES [17]
is executed for fixed generations and the
CMA-ES phase is then applied. SaDE
[24] summarizes historical information
during the search procedure and uses it
to adaptively select operators from a pool
of mutation operators in each generation.
cDE [40] defines a candidate pool
of operators and selects the best one
based on the number of success of each
operator. MPEDE [34] divides the population
into four parts, and assigns three
of them to three DE operators and the
fourth to the operator that delivers the
best performance in the previous several
generations. The notations used in this
paper are listed in Table I.
B. Learning-to-Optimize Technology
The deep neural network is typically
used in the implementation of learningto-optimize
to represent the knowledge
learned for optimization. Andrychowicz
et al. [14] proposed learning the descent
EXPLANATION
the action at the t-th time step
the policy with parameter u
the transition probability
the state-value function
the action-value function
deep Q network with parameter w
the minimum and initial population size
the parameter used to decide when to use LPSR
the original LSHADE
the LSHADE with HPSS
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
