IEEE Computational Intelligence Magazine - May 2023 - 22

FIGURE 1 The proposed multitasking framework for CMOPs.
FIGURE 3 Illustration of the proposed QL-based evolution.
The actions are performing one ofthe auxiliary tasks. They
include
A¼ aja 2 T1; T2; ... ; Tifg
fg;
(1)
where T1 to Ti represents performing the corresponding task, and
i is the number ofauxiliary tasks. The states are the currently used
auxiliary tasks. They include
S¼ sjs 2ðT; T1Þ; ðT; T2Þ; ... ; ðT; TiÞfg (2)fg:
FIGURE 2 The proposed Q-Table.
strategies are suitable for different CMOPs. Since real-world
problems are usually subject to unknown features, it is necessary
to develop approaches to adaptively select the best strategy
when dealing with a given problem. Moreover, the dynamic
evolutionary process requires an approach to select the strategy
according to the population state adaptively. The following
section explains the proposed methods in detail.
III. Our Approach
A. Proposed EMTFramework
Our EMT framework is inspired by the EMT methodology
and the existing CMOEAs using helper problems to assist in
solving the original CMOPs as shown in Figure 1.
The proposed framework contains a main task for the original
CMOP and one or more auxiliary tasks for helper problems.
Any existing or newly established algorithmic strategy can be
embedded as the strategy ofthose auxiliary tasks. Moreover, solutions
of both the parent and the offspring sets are regarded as
knowledge to avoid labor-intensive transfer strategy design.
Consequently, the proposed framework can provide a foundation
for the auxiliary task selection. Additionally, the framework
has good expansibility due to its simple structure. This framework
employs the RL techniques to select the task adaptively.
The QL andDQLmethods are implemented as follows.
B. Proposed Q-Learning Model and CMOQLMT
In QL, the states, actions, and rewards should be set so the
agent can learn the Q-Table and make decisions accordingly.
This work defines these definitions as follows.
22 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2023
For example, ðT; T1Þ means the algorithm currently adopts T1
as the auxiliary task. The reward ofperforming thej-th task Tj
is defined by the transfer ratio from TJ to T:
jPt
rt ¼
jj
jPjj
;
where Pj is the population ofTj; and Pt
(3)
j is the set of solutions
transferred to T (i.e., selected into T for the next generation).
Then, the Q-Table is updated using the Bellman function
Qðs; aÞ¼ Qðs; aÞþ a½r þ gmaxa0Qðs;0 a0ÞQðs; aÞ;
(4)
where a is the learning rate, and g is the discount factor. As the
core idea of QL, the Bellman function considers both the realtime
reward ofperforming action a under state s and the expected
future reward of taking the next action. Figure 2 shows the
Q-Table of the proposed model. This Q-Table can estimate the
Q-values ofperforming any auxiliary task under any state.
Based on this model, the proposed QL-based evolution
can be illustrated in Figure 3. The evolutionary process contains
four parts.
❏ Evolution: In the evolution process, the useful knowledge
(solutions) in the selected auxiliary task is transferred to the
main task to assist in the evolution ofthe main task.
❏ Interaction: In the interaction process, the agent performs
the action (i.e., transfer knowledge from the selected auxiliary
task) and determines the reward and the next state.
❏ Learning: The learning process updates the Q-Table
according to the results ofInteraction.
❏ Decision: The decision process determines the action
with the maximum expected reward based on Q-Table
and the most expected auxiliary task.
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
