IEEE Computational Intelligence Magazine - February 2023 - 84

Research
Frontier
Haotian Zhang and Jianyong Sun
Xi'an Jiaotong University, CHINA
Thomas B€ack
Leiden University, THE NETHERLANDS
Qingfu Zhang
City University ofHong Kong, HONG KONG
Zongben Xu
Xi'an Jiaotong University, CHINA
Controlling Sequential Hybrid Evolutionary
Algorithm by Q-Learning
Abstract
M
any state-of-the-art evolutionary
algorithms (EAs) can
be categorized as sequential
hybrid EAs, in which various EAs are
sequentially executed. The timing to
switch from one EA to another is critical
to the performance of the hybrid EA
because the switching time determines the
allocation of computational resources and
thereby it helps balance exploration and
exploitation. In this article, a framework
for adaptive parameter control for hybrid
EAs is proposed, in which the switching
time is controlled by a learned agent rather
than a manually designed scheme. First the
framework is applied to an adaptive differential
evolution algorithm, LSHADE, to
control when to use the scheme to reduce
the population. Then the framework is
applied to the algorithm that won the
CEC 2018 competition, i.e., the hybrid
sampling evolution strategy (HSES), to
control when to switch from the univariate
sampling phase to the Covariance
Matrix Adaptation Evolution Strategy
phase. The agents for parameter control in
LSHADE and HSES are trained by using
Q-learning and deepQ-learning to obtain
the learned algorithms Q-LSHADE and
DQ-HSES. The results ofexperiments on
the CEC 2014 and 2018 test suites show
that the learned algorithms significantly
Digital Object Identifier 10.1109/MCI.2022.3222057
Date ofcurrent version: 13January 2023
IMAGE LICENSED BY INGRAM PUBLISHING
outperform their counterparts and some
state-of-the-art EAs.
I. INTRODUCTION
Evolutionary computation has been studied
since the 1950s [1] and many promising
evolutionary algorithms (EAs) have
been proposed for solving black-box optimization
problems, such as the genetic
algorithm (GA), [2] differential evolution
(DE), [3], [4] particle swarm optimization
(PSO), [5] evolution strategies (ES), [6]
and evolutionary programming (EP) [7].
The successful application of an EA
depends on a variety of factors, including
but not limited to the incorporation of
prior knowledge ofthe optimization problem
at hand, the design ofthe algorithmic
components (including the recombination
and selection operators), and the determination
ofthe algorithmic parameters. The
Corresponding author: Jianyong Sun (e-mail:
jy.sun@xjtu.edu.cn).
84 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
algorithmic parameters of an EA significantly
influence its performance. In the
previous work [8], these algorithmic parameters
were categorized as either structural
or numerical. The structural parameters of
an EA control the algorithmic procedure
and hence influence the computational
complexity of the algorithm. Taking a
hybrid EA as an example, when to switch
from one EA phase to another is an important
structural parameter. Parameters such
as the scaling factor (F) and the crossover
rate (CR) in DEs, and the crossover and
mutation probability in GAs can be classified
as the numerical parameters, which
are usually directly responsible for the generation
ofoffspring.
The process of finding the optimal
time-invariant parameters ofEAs is commonly
referred to as " parameter tuning "
[9]. To find the optimal parameters, the
" parameter response function, " i.e., the
metric that measures the performance of
the considered EA, is usually optimized.
Considering the intrinsic randomness of
the EA, its performance should be measured
by executing it several times for
each set of parameter configurations,
which is highly time consuming. Furthermore,
information on the derivatives
ofthe parameter response function is usually
unknown. These factors make
parameter tuning an expensive optimization
problem [9].
1556-603X ß 2023 IEEE
https://orcid.org/0000-0001-7395-1147 https://orcid.org/0000-0002-9188-1856 https://orcid.org/0000-0001-6768-1478
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
