IEEE Computational Intelligence Magazine - May 2023 - 25

FIGURE 7 The general framework of the proposed CMODQLMT algorithm.
not only calculate the current reward for performing action a,but
also estimate the maximum cumulative reward in the future.
Based on the DQL model, this work develops the DQLbased
evolution process shown in Figure 5. The evolution
process ofthe DQL-based evolution is similar to the QL-based
evolution. However, in the interaction process, the feedback is
a record t; the Network is trained in the learning process; each
action is adopted as the input of the trained Network in the
decision process to obtain the Q-values. Then, the action with
the maximum Q-value is selected for transfer.
Since the procedure ofCMODQLMT is similar to that of
CMOQLMT, the pseudo-codes of the " Framework " and the
" SelectAction " are placed in the supplementary file to save space.
This work designs the Network structure as shown in
Figure 6. The input of the Network includes the population
state at the generation when employing the Network and the
action. The S1; S2; S3 representfit;f;f, respectively. A normalized
encoding is performed on the input to eliminate the influence
of different scales on the accuracy ofthe Network before
employing the Network for prediction. The action is marked as
A. The Network contains two hidden layers and dropouts. It
employs two hidden layers so dropouts can be applied, and the
time cost, as well as the complexity of the structure, can be
reduced. The output of the Network includes the Q-value of
performing the action A and the new state including S1;0 S2;0 S0
3
representing thefit;f;f ofthe new population after performing
A. Afterwards, a reverse decoding is applied to the output to
regress the data mapping to the original scales as the final output.
Dropouts are adopted for the following two reasons. First,
dropouts can reduce the training time and alleviate the extra
time cost ofthe DQL technique. Second, they can enhance the
generalization ofthe Network and prevent overfitting, which is
suitable for evolutionary algorithms since they can usually provide
only small data for training due to limited function evaluations.
The ReLU function is adopted as the activation function
for the hidden layers because of its minimum computational
cost and promising effect in a simple network structure.
CMODQLMT is formed based on the DQL model and
Network. The procedure is shown in the flowchart in Figure 7.
It is generally similar to that ofFigure 4, except for the following
differences.
❏ Since DQL needs an EP to provide historical records for
training the Network, the auxiliary task is randomly
selected in CMODQLMT until the EP reaches the
required size (i.e., full). It could enhance the exploration in
the early stage.
❏ If the EP is full and the Network is not built, the Network
is constructed before using it to select the auxiliary task.
❏ After constructing the Network, it updates every 50 generations
to enhance its effectiveness and avoid the time cost
offrequent training.
D. Instantiation ofCMOQLMTandCMODQLMT
The proposed CMOQLMT and CMODQLMT provide two
frameworks for using QL and DQL to learn to select the auxiliary
task, resulting in two instantiations with the following four
existing algorithmic strategies. Researchers could employ
other existing strategies or devise new strategies to construct
CMOQLMT and CMODQLMT.
❏ The main task: The main task T aims to solve the original
CMOP. Therefore, SPEA2 that employs the constraint dominance
principle (CDP) as CHT is adopted for T. SPEA2CDP
is a widely used algorithmic strategy in CMOEAs [6],
[7], [21], where its effectiveness in balancing convergence
and diversity has been empirically demonstrated.
❏ The first auxiliary task: The penalty-based strategy of cDPEA
[9] is adopted.
❏ The second auxiliary task: The Pareto dominance (PD)-based
SPEA2 [42] strategy that ignores constraints is adopted.
❏ The third auxiliary task: The even-search-guided strategy of
ICMA [10] is adopted.
IV. Experimental Results and Analysis
This section gives the experimental results and analysis. All
experiments were conducted on the PlatEMO [43].
A. Experimental Settings
1)Test Problems
The experimental studies employed five benchmark CMOP test
suites and one real-world CMOP test suite to evaluate the proposed
methods on both the tailored and the real-world problems.
The five benchmarks include C-DTLZ [44],DCMAY
2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 25
IEEE Computational Intelligence Magazine - May 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2023
