IEEE Computational Intelligence Magazine - February 2023 - 85

Time-variant (or adaptive) parameter
control has been more popular in recent
work on EAs than the use oftime-invariant
parameters. Parameter control means
adaptively setting the parameters ofan EA
during the evolutionary search process.
This has mostly involved the adaptive
control ofnumerical parameters, such as F
and CR in DE, [10], [11], [12] in the literature.
By contrast, few studies have considered
controlling such structural
parameters as the population size [13].
Further, almost all adaptive mechanisms
have been designed heuristically, which
may restrict the capacity of the control
mechanism because finding the optimal
mechanism is challenging and time-consuming,
and the ability of humans for
searching is limited. In this paper, the
search for the optimal control mechanism
is modeled as a " learning " problem based
on the recently proposed " learning-tooptimize "
technology [14], [15].
The basic idea of " learning-to-optimize "
is to learn useful knowledge by
optimizing related problems and using
this knowledge to optimize new problems
efficiently. In this paper, we propose
applying this idea to design an adaptive
mechanism for structural parameters in
sequential hybrid EAs. A sequential
hybrid EA is composed of several phases
in a combination ofEAs. The underlying
idea is to use the advantages of different
EAs in different stages ofsearch to ensure
excellent algorithmic performance. The
timing to switch from one phase to
another is thus critical for performance.
Due to the randomness of the search
process of an EA, one way to find the
optimal adaptive mechanism to determine
the switching time is via reinforcement
learning (RL),which is based onmodeling
thesearch process of theEAasaMarkov
decision process (MDP). In this paper, we
propose training an agent to control the
switching time by using two popular RL
algorithms: Q-learning and deep Q-learning
[16]. Our contributions can be summarized
as follows:
❏ An RL-based framework for
sequential hybrid EAs is proposed,
for the first time in the literature, in
which the switching time is controlled
by an RL agent that learns by
using Q-learning/deep Q-learning.
An RL-based framework for sequential hybrid EAs is
proposed, for the first time in the literature, in which
the switching time is controlled by an RL agent that
learns by using Q-learning/deep Q-learning.
❏ The framework is applied to the
HSES [17] and a modified version
ofLSHADE, and the resultant algorithms
are called DQ-HSES and QLSHADE.
❏
The proposed algorithms are evaluated
by comparing them with three wellknown
EAs on the CEC 2014 and
2018 test suites. The results show that
DQ-HSES and Q-LSHADE can significantly
outperform their counterparts,
i.e., the HSES and LSHADE,
respectively. Further, the results show
that DQ-HSES outperforms the compared
EAs in general, which implies
that the learned RL agent can significantly
improve the performance of
existing sequential hybrid EAs.
The remainder of this paper is organized
as follows: Section II brieflyintroduces
related work, including studies on
parameter tuning/control, and provides
the preliminaries on learning-to-optimize
and reinforcement learning. Section III
details the proposed Q-learning-based
framework for the adaptive control ofthe
structural parameters. Its applications to
LSHADE and HSES are presented in
Sections IV and V, respectively.
Sections VI and VII show the experimental
results obtained on the CEC 2014 and
2018 test suites, and Section VIII summarizes
the conclusions ofthis paper.
II. RELATED WORKAND
PRELIMINARIES
A. Parameter Tuning and Control
1) Parameter Tuning
Parameter tuning refers to the process of
choosing a set of optimal parameters for
an EA. Derivative-free methods ofoptimization,
such as the Bayesian Optimization
Algorithm (BOA) [18], Sequential
Model based Algorithm Configuration
(SMAC) [19], and Parameter Iterative
Local Search (ParamILS) [20], are often
used for parameter tuning in EAs.
Roman et al. [21] used the BOA to tune
the parameters including the spread
parameter, the population size, and the
offspring size in their hybrid kernel estimation
of the distribution algorithm.
Huang et al. [22] used the BOA to tune
the numerical parameters ofan EA called
the scalable approach based on hierarchical
decomposition [23].
2) Parameter Control
Three mechanisms are mainly used to
control numerical parameters. First a
numerical parameter is generated/sampled
randomly in each generation. In
SaDE [24], for example, F is sampled
from a Gaussian distribution Nð0:5; 0:3Þ
in each generation for each individual. In
SaNSDE [25], F is set based on either a
fixed normal distribution or a Cauchy
distribution.
In the second mechanism to control
numerical parameters, the parameter is
generated adaptively based only on information
collected during the evolution
process. In JADE proposed by Zhang
et al. [10], the scaling factor F (resp., the
crossover rate CR)is generated from a
Cauchy (resp., Gaussian) distribution in
which the median (resp., mean) is the
Lehmer mean (resp., arithmetic mean) of
successful parameters in the last generation.
Some well-known DEs, such as
LSHADE [12], iL-SHADE [26], andjSO
[27], have the same mechanism as JADE.
In the adaptive PSO proposed by Zhan
et al. [28], the inertia weight and the
acceleration coefficients are adaptively
generated according to information collected
during the evolution process.
In the third mechanism, the numerical
parameters are controlled by both a
handcrafted mechanism and information
collected during the evolution process. In
jDE [29], F and CR are either inherited
from the last generation or sampled from
pre-fixed uniform distributions for each
individual. CoBiDE [30] samples F and
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 85
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
