IEEE Computational Intelligence Magazine - May 2023 - 24

The pseudo-code ofthe selection ofactions is presented in
Algorithm 2. A greedy threshold " is used to control whether
the action is randomly selected or based on the Q-Table.
When selecting based on the Q-Table, the auxiliary task Tj
with the maximum expected reward under the current state
ðT; TcÞ is selected.
C. Proposed Deep-Q-Learning Model and CMODQLMT
The core issue in establishing the DRL model is to determine
the states, actions, and rewards, so the agent can learn the relations
between population states, actions, and rewards through
a deep neural network. This work employs the average fitness
fit , the average sum of objective function valuesf, and the
average constraint violation (CV) valuesf of solutions in the
population to represent the state. Since the original CMOP is
the final target, only the population state of the main task T
needs to be estimated.
The average fitnessfit is formulated as
fit ¼
P
x2PfitðxÞ
N
;
(5)
where P is the population ofa task;fitðxÞ is the fitness value of
solution x; and N is the population size. It should be noted
that since the original CMOP is related to the main task, the
fitness values are obtained by the adopted algorithmic strategy
ofthe main task T. The algorithmic strategy ofa CMOEA for
the original CMOP, in general, should consider the optimization
ofconvergence, diversity, and feasibility performance.
The average sum of objective function value is adopted to
represent the position ofthe population in the objective space,
i.e., the distance between the population and the CPF. Specifically,
it is formulated as
f ¼
P Pm
N
x2P
j¼1fjðxÞ
;
(6)
wherefjðxÞ is thej-th objective function value.
The average CV value is adopted to estimate the feasibility
performance and the position ofthe population in relation to the
feasible and infeasible regions. Specifically, it is formulated as
f ¼
P
x2P fðxÞ
N
;
(7)
where fðxÞ is the overall CV ofx.
Then, the above three components are adopted to form
the state. The state set is
S¼ sjs ¼ðfit;f;fÞfg:
(8)
DQL requires an experience replay (EP) set to save historical
records for training the neural network. This work designs
the record as
t ¼fg;
fit ;f;f; a; rt;fit;0f;0f0
24 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2023
(9)
FIGURE 6 Illustration of the proposed DQL Network.
FIGURE 5 Illustration of the proposed DQL-based evolution.
where T is the training data; Qðst; atÞ is the output ofthe Network
with the input ðst; atÞ, and qt is the Q-value oftaking the
action at at state st formulated as
qt ¼ rt þ gmaxa02AQðstþ1; a0Þ;
(12)
where Qðstþ1; a0Þ is the maximum reward for taking the next
action under the state at which at is performed. Therefore, qt can
where the former three components consist of the population
state; a is the action (i.e., the selected auxiliary task); rt is the
reward calculated by (3), and the last three components comprise
the new state. The EP T is designed as a queue
T¼ t1; t2; ... ; tsz
fg;
(10)
where sz is the maximum size ofEP. The oldest record will be
deleted based on the first-in, first-out rule when the number
ofrecords exceeds the maximum size.
The input of the deep Q-Network (named Network) is
the state of the population and the action (i.e., ðs; aÞ), and the
output as the Q-value of the action a and the new state of the
population. Therefore, the DQL designed in this work adopts
the Network to estimate the reward of an action. With the
records ofT, the Network is trained using the gradient descent
on the following loss function:
L¼
1
jTj
X
t2T
ðQðst; atÞ qtÞ;
(11)
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
