IEEE Computational Intelligence Magazine - February 2023 - 102

Thisshowsthatthe switcher
agent
can better allocate computational
resources without sacrificing too
much time.
FIGURE 5 The visualization of the results of voting for 10D (a), 30D (b), 50D (c), and 100D
(d) functions. Red regionmeans not switching to CMA-ES; Yellowregionmeans switching to
CMA-ES; Blue regionmeans selecting a randomaction.
TABLE XII The average CPU times of HSES and DQ-HSES (in seconds).
10D
4.06
5.77
30D
HSES
DQ-HSES
31.32
32.99
50D
99.83
98.46
100D
608.79
623.31
VIII. CONCLUSION
In this article, we propose a framework
for the adaptive control of the structural
parameters of sequential hybrid EAs
based on Q-learning. By way of case
studies, we apply the framework to
adaptively control the switching time
of the strategy to set the population
size in LSHADE as well as that of
HSES from the univariate sampling
phase to the CMA-ES phase. The
experimental results on the CEC 2018
test suite show that the learned algorithm
based on LSHADE, QLSHADE,
outperforms LSHADE. The
learned algorithm based on HSES (the
winner of the CEC 2018 competition),
DQ-HSES, also outperforms it as well
as some well-known EAs. The results
verify the effectiveness of the proposed
framework and indicate that learned
knowledge is useful for solving new
optimization problems. In the future
work, we intend to study how to
adaptively design EAs instead of only
controlling their algorithmic parameters.
Moreover, we will examine ways
to learn local search algorithms to solve
combinatorial optimization problems.
on the CEC 2018 test suite excluding
the training functions. It is clear that
CoBiDE obtains the minimum APS on
10D functions, and DQ-HSES obtains
the minimum APS on 30D and 100D
functions. The last two rows ofthe table
show the average APS and the
weighted-average APS on the functions
over the four sets of dimensions with
weights 0:1; 0:2; 0:3; and 0:4 for 10D,
30D, 50D, and 100D functions, respectively,
as used in the CEC 2017 competition
[27]. It is clear that DQ-HSES
achieves the minimum average and
weighted-average APS values. Moreover,
Q-HSES has a higher score than
HSES but lower than DQ-HSES,
which implies that using Q-learning
can improve the performance of HSES
while DQN can deliver even better
results. Thus we may conclude that
DQ-HSES is competitive with the
state-of-the-art EAs.
4) Visualization ofDQNs
Figure 5 shows the results of voting of
DQNs on four dimensions. It shows the
following: 1) The results of voting are
different in different dimensions. 2) For
each dimension, the result is too irregular
to be manually designed. This verifies
the advantage ofusing a trained agent.
5) Time Complexity Analysis
Table XII shows the average running
times of HSES and DQ-HSES on
F1 F30 for each set of dimensional
functions. It is clear that DQ-HSES
generally requires slightly longer
than HSES except on 50D problems.
102 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
Acknowledgment
This work was supported in part by the
National Natural Science Foundation
of China under Grants 11991023,
62076197, and 61903294, in part by the
Major Project ofNational Science Foundation
ofChina under Grant U1811461,
in part by theKey Project ofNational Science
Foundation of China under Grant
11690011, and in part by the Project of
National Science Foundation of China
under Grant 61721002.
References
[1] T. B€ack, Evolutionary Algorithms in Theory and Practice:
Evolution Strategies, Evolutionary Programming,
Genetic Algorithms. New York, NY, USA:Oxford
Univ. Press, 1996.
[2] D. E. Goldberg, Genetic Algorithm in Search, Optimization,
and Machine Learning. Reading, MA, USA:
Addison-Wesley, 1989.

IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023