IEEE Computational Intelligence Magazine - February 2023 - 42

accommodate new tasks that are completely different from the
previous ones. The model introduces a new measurement to
regulate the evolution in graph data with different characteristics
to address that objective. The method has the capability to
integrate an isotropic and anisotropic type [87] ofGNN.
The problem definition ofthis model is as follows. Assuming
graph data has a finite number of tasks T1; ...; Tt and a
modelf with parameters u. To do incremental training in this
method, while the structure of the graph including its nodes
and edges is changing over time, during preparation for task
Ttþ1, a model should have been trained based on the label on
task Tt to get the uðtÞ parameters. When l new classes come,
these methods add the corresponding number ofparameters to
the output layer offðtÞ. Therefore it will have a new parameterized
layer:
j ut
j ut
output weight j¼jut1
output bias j¼jut1
output weight jþ l; and
output bias jþ l:
(3Þ
(4)
In the next step, to enable the rehearsal approach, this
method gives the option to choose whether to retrain the
model from scratch (cold restart) or use the final model parameters
from the previous tasks (warm restart). The model
involves an artificial history size that determines the amount of
past data to control against full-history retraining.
Since the pre-compiled dataset for graph lifelong learning
is rare, this model performs an experiment on the scientific
publication dataset PharmaBio and DBLP (easy and hard) [88].
The scenario is to learn new tasks based on adding new classes.
Two evaluation measures used average accuracy and forward
transfer to measure the effect ofreusing previous parameters. It
also uses several graph neural networks technique to perform
representation learning in the model, such as GAT [78],
GCN [27], and graphSage [75]. The main result shows that
incremental learning using limited history sizes is almost similar
to the default setting by using the full history size ofthe graph.
GNN technique in this model achieves at least 95% accuracy
that can be retrained with half dataset coverage compared to
using all past data for incremental learning.
V. REGULARIZATION APPROACH
This approach implements a single model and has a fixed
capacity by leveraging the loss function using the loss term to
help consolidate knowledge in the learning process for new
tasks and retain previous knowledge [19], [20]. Prior knowledge
of graph structures and tasks will be maintained to
achieve stable performance while learning novel knowledge.
In graph-based learning, some current works that implement
regularization approaches are DiCGRL [68], GPIL [69],
TWP [70], and Translation-based Knowledge Graph Embedding
[71]. Other methods that partially implement a regularization
approach in graph data that are explained in more detail
in Section VI are ContinualGNN [10], LDANE [72], and
TrafficStream [44].
42 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
A. Disentangle-Based Continual Graph Representation
Learning
Kou et al. [68] proposed Disentangle-based Continual Graph
Representation Learning (DiCGRL), which focuses on applying
lifelong learning in graph embedding. Current graph
embedding models are not practical in real-world applications
because they tend to ignore streaming and incoming data
nature in graphs. DiCGRL takes two approaches to continuously
learn a new graph embedding whilst preventing the forgetting
of old knowledge. First, the disentangle module aims
to convert the graph's relational triplets (u; r; v), which explains
the connection between node u and v based on relation r into
several components in their semantic aspects. The model uses
an attention mechanism to learn the attention weight ofcomponents
u and v based on the relation r. Then, top-n-related
components are processed through a graph embedding mechanism
that incorporates the features ofnodes u and v. There are
two types of graph embedding used to do this: knowledge
graph embedding (KGEs) [89], [90] and network embedding
(NEs) [78], [91].
The second step is updating the module that aims to
update disentangled graph embedding when new relation triplets
appear. This model uses a regularization approach to prevent
the forgetting problem. In this process, a step called
neighbor activation identifies which triplet relation should be
updated by running a selection mechanism inspired by the
human ability to learn procedural knowledge. Top-n components
with common characteristics are selected to update the
embedding representation. Then, when the new data arrive,
the incoming data will be trained with those activated neighbor
components using a constraint in loss function to accommodate
new knowledge and maintain the performance of
prior tasks. DiCGRL employs constraint loss terms Lnorm to
push the sum of attention weights of top-n selected components
reaching 1, i.e.,
Lnorm ¼
ðu;r;vÞTi
X
1
Xn
k
ak
r
;
(5)
where n indicates the number of components. So that the
overall loss function to implement the regularization process in
this method is as follows:
L¼Lold þLnew þ b:Lnorm;
(6)
where b is a hyperparameter.
One of the techniques to measure the performance of
DiCGRL is to use link prediction tasks and node classification
tasks. For link prediction tasks, it uses two datasets, FB15K237
[92] and WN18RR [93], that are set to have instances
that come sequentially. Moreover, two types of knowledge
graph embedding, such as TransE [90] and ConvKB [94], are
used to perform graph representation learning. For node classification
tasks, It uses several datasets such as Cora, CiteSee, and
Pubmed [75] with the scenario of adding instances sequentially.
There are some strategies used to compare the
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
