IEEE Computational Intelligence Magazine - May 2023 - 23

FIGURE 4 The general framework of the proposed CMOQLMT algorithm.
Algorithm 1. The Framework of CMOQLMT
Require: N (population size), Gmax (termination condition), i
(number of auxiliary task)
Output: P (final solution set)
1: P Generate initial population for the main task T;
2: P1; .. . ; Pi Generate initial populations for auxiliary tasks
T1; .. . ; Ti;
3: Q Initialize the Q-Table as zeros;
4: g 0;
5: while g < Gmax do
6:
if g < l Gmax then
7: O Generate offspring set of T;
8:
9:
12:
13:
16:
19:
20:
21:
O1; .. . ; Oi Generate offspring set of T1; .. . ; Ti;
Tj Select an action according to Algorithm 2;
10: P Transfer solutions of Tj to T and evolve T;
11:
Q Update Q-Table according to (4);
P1; .. . ; Pi Evolve T1; .. . ; Ti;
else
14: O Generate offspring set of T;
15:
Tj Select an action according to Algorithm 2;
Oj Generate offspring set of Tj;
17: P Transfer solutions of Tj to T and evolve T;
18:
Q Update Q-Table according to (4);
Pj Evolve Tj;
end if
g g þ 1;
22: end while
23: return P
Based on the QL model and the QL-based evolution process,
this work designed a new method called CMOQLMT.
Its procedure is shown in Figure 4, which uses different colors
to distinguish processes of different purposes. Green, blue, and
yellow colors represent the evolution of auxiliary tasks, the
evolution ofthe main task, and the QL-based learning process.
Generally, the algorithm includes the learning and evolving
stages. In the learning stage, all auxiliary tasks evolve (i.e., generate
offspring and perform environmental selection) for two
goals. All auxiliary tasks evolve so they can converge according
to their algorithmic strategies. Otherwise, the positions oftheir
populations might be far from the CPF or the UPF, resulting in
useless transfer. Besides, they evolve to enhance the exploration
of QL. In contrast, only the selected auxiliary task evolves in
the evolving stage for the following consideration. In the later
stage ofevolution, the function evaluations should be concentrated
on the better-suited auxiliary task to utilize the learned
policy to enhance the exploitation.
The learning stage first generates the offspring sets of all
auxiliary tasks. Then, the algorithm selects one ofthe auxiliary
tasks based on the Q-Table. Knowledge of the selected auxiliary
task is transferred to the main task T, and the environmental
selection is performed to select the population ofT for the
next generation. Afterwards, the Q-Table is updated based on
the feedback of interaction. Then, all auxiliary tasks perform
their environmental selections to evolve.
The evolving stage selects the auxiliary task (assumed Tj)
based on the Q-Table. Then, the offspring set ofTj is generated,
and the knowledge (useful solutions) is transferred to T to help
the environmental selection ofT.Afterwards, Tj evolves and the
Q-Table is updated. After terminating the algorithm, the population
ofT is output as the final solution set.
The pseudo-code ofthe proposed CMOQLMT is presented
in Algorithm 1. The processand stepsare thesameasFigure 4.
P; P1; .. . ; Pi are the population sets for corresponding tasks. QTable
is represented by a matrix Q structured as Figure 2.The
parameter l is used to control the length ofthe learning stage.
Algorithm 2. Select Action
Require: " (greedy threshold), ðT; TcÞ (current state), A (action
set), i (number of auxiliary tasks), Q (trained Q-Table)
Output: Tj (selected action)
1: k Generate a random number in [0,1];
2:
if k " then
3:
Tj ¼ argmaxa2A QððT; TcÞ; aÞ;
4: else
5:
j Generate a random number in f1; 2; .. . ; ig;
6: end if
7: return Tj
MAY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 23
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
