IEEE Computational Intelligence Magazine - May 2023 - 19

prevalence of CMOPs in real-world problems, such as the
vehicle routing problems [1], the testing resource allocation
problems [2], the vehicle scheduling of the urban bus line [3],
and the web service location-allocation problems [4].1
Different constrained multi-objective optimization evolutionary
algorithms (CMOEAs) [5] have been proposed,
where most of them employ one or more fixed algorithmic
strategies mainly to solve helper problems for the original
CMOPs. For example, some adopted the Pareto
dominance-based strategy to solve an unconstrained helper
problem [6], [7]; others employed the constraint relaxation-based
strategies to solve a constrained-relaxed helper
problem [8], [9]; and still others applied the infeasible solution
preservation strategies to solve an even-search-guided
helper problem [10], [11]. Although these different strategies
can be well-applied to the benchmark CMOPs with
specific features and challenges, they may encounter the
following difficulties when solving real-world CMOPs subject
to unknown features and challenges.
❏ Since the distribution of solutions in the population
changes dynamically [12], the most expected and helpful
algorithmic strategy could vary over different stages, rather
than be fixed.
❏ A specific algorithmic strategy is suitable for CMOPs with
some features and challenges but exhibits poor performance
on others [13]. However, determining the most suitable
one is very difficult and usually problem-dependent.
In order to tackle the first difficulty, as pointed out in [12],
reinforcement learning (RL) techniques can learn a policy that
helps the agent choose the most suitable action in a dynamic environment.
As the candidate algorithmic strategies are regarded as
actions, RL can ideally learn to adaptively select the most suitable
one at any moment ofthe dynamic evolutionary process. Therefore,
adopting RL techniques to overcome the first difficulty is
very promising. The ensemble has been widely employed to solve
the second difficulty [14]. Intuitively, using auxiliary algorithmic
strategies (usually through solving helper problems) to solve the
original CMOP is similar to the evolutionary multitasking (EMT)
methodology [15], [16].EMT assistsinsolving atargetproblem
through knowledge transfer between different tasks (usually from
source/auxiliary tasks to the target tasks). Consequently, EMT can
achieve the ensemble ofalgorithmic strategies taking different auxiliary
algorithmic strategies as source tasks and handling the original
CMOP as the target task.
Based on these considerations, this work proposes an adaptive
auxiliary task selection method based on RL techniques
for CMOPs. The main contributions ofthis work can be summarized
as follows:
1) A general multitasking framework is proposed to transform
the original CMOP into a multitasking optimization problem.
It includes a main task for the original CMOP and
auxiliary tasks for helper problems. An arbitrary number of
1.The basic concepts ofCMOP are introduced in the supplementary file.
Afixed/specific algorithmic strategy
cannot handle every CMOPwell
according to the No Free Lunch theory.
Therefore, ensemble is an effective
approach to make use of the pros and
alleviate the cons of different algorithms.
auxiliary tasks based on different strategies can be embedded
into the framework to solve the CMOPs.
2) The RL techniques are implemented in the framework to
choose the most suitable auxiliary task adaptively. As
instantiations, the Q-Learning (QL)-based and Deep-QLearning
(DQL)-based auxiliary task selection methods are
designed. With the help of QL and DQL techniques,
selecting the most suitable auxiliary task considers not only
the historical experience ofevolution but also the influence
ofthe strategy in the future state.
3) Based on the proposed framework, two new algorithms,
CMOQLMT and CMODQLMT, are derived. The proposed
methods include learning stage and evolving stage to
balance exploration and exploitation. To ensure the exploration,
all auxiliary tasks evolve in the learning stage; in
addition, the most suitable auxiliary task for knowledge
transfer is selected according to the Q-Table or Q-Network
in this stage. In the evolving stage, only the most suitable
auxiliary task evolves, and only the knowledge of this
task is transferred to strengthen the exploitation.
In order to evaluate the performance of the proposed
methods, they were compared with 11 advanced CMOEAs
on five benchmark problem test suites and 28 real-world problems
from the IEEE CEC 2021 Competition [17]. The results
reveal that our approaches can handle different benchmark
CMOPs well and achieve superiority over other CMOEAs on
the real-world CMOPs on the whole.
It should be noted that although some research works in
machine learning also named " auxiliary task selection " [18], [19],
they mainly focus on using different auxiliary tasks to assist the
training ofRL, rather than solving optimization problems.
The rest of this paper is organized as follows. Section II
briefly introduces the related work. The proposed methods are
elaborated on in Section III, followed by the experimental
results and discussions in Section IV. Finally, the conclusions
and future work are presented in Section V.
II. Related Work
A. Existing Algorithmic Strategies for CMOEAs
In the past decade, many research efforts have been dedicated
to developing CMOEAs [5]. The existing CMOEAs can be
divided into three categories based on the employed strategies
for solving helper problems.
MAY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 19
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
