IEEE Computational Intelligence Magazine - February 2023 - 44

regularizing the loss function while learning a new task.
When changing high-importance parameters is penalized,
themodel canstill performwellonprevioustasks. The loss
function for new tasks Tkþ1 formulated as:
L0
kþ1ðWÞ¼ Lnew
where Lnew
kþ1 þ
Xk
n¼1
InðWW
n Þ2;
(7)
kþ1 is a new loss function, e.g., cross-entropy based on
the task. In denotes the network importance for the old tasks.
means element-wise multiplication and W
n
contains the
optimal parameter for task Tn. That learning strategy ensures
that low-important score parameters can be adjusted freely
while high-importance parameters will be maintained and
penalized so that the model can handle the previous tasks.
TWP also employs a strategy to promote the minimization of
the computed importance score to maintain the model
plasticity.
One ofthe experiments on this method uses a node classification
case study. Some datasets are used on node classifications,
such as Corafull [100], amazon computer [97], protein-protein
interaction (PPI) [4] and Reddit [35]. Three backbones of
GNN are used to perform representation learning, such as
GAT [78], GCN [27], and GIN [80], which are connected
using several general lifelong learning such as LWF [20],
GEM [23], EWC [19], etc. TWP got the highest average performance
with 0.976 and got the lowest average forgetting
score of 0.001 0.062 on the Reddit dataset. TWP demonstrates
the effectiveness and ability to perform lifelong learning
in the graph domain compared to other standard lifelong learning
methods.
D. Translation-Based Knowledge Graph Embedding
Song and Park [71] developed a translation-based knowledge
graph embedding method through continual learning that
aims to enrich the representation of knowledge graphs when
incoming representation appears in graph data. Moreover, this
method minimizes the risk over new triples using rapid parameters
that are penalized between the old and new embedding
models. The translation-based knowledge graph embeddings
aim to generate a vector representation in the embedding
space of the knowledge graph based on the entity relation
information in a graph. The simplest model in the translationbased
knowledge graph embedding is TransE [90], which
forms the embedding matrix of entities and relations in the
graph. In that case, the parameters to create the embedding are
optimized using stochastic gradient descent.
The assumption is that the knowledge graph embedding
method [89], [90] is those graph entities whose relations are
fixed after the training. This method employs the regularization
term in order to accommodate the new embedding representation
and preserves the representation of previously
generated embedding. The underlying concept ofthe regularization
approach in this model is to minimize empirical risk
Rn
empðunÞ defined as:
44 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
Rn
empðunÞ¼ Rn
where Rn
empðDn; unÞþnfðun; un1Þ;
(8)
empðDn; un) denotes the empirical risk of new data.
is a hyperparameter that maintains
fðun; un1Þ is as regularization term to penalize new parameter
so that un close to un1.n
how much un retrains the knowledge from previous data. As a
result, the method can achieve the objective by minimizing
risk penalized by rapid representation changes, and the old
experiences are preserved in a new model.
One of the experiments with this method is in a knowledge
graph embedding scenario with two knowledge graph
datasets: WN18 (WordNet) [93] and FB15 k (Freebase) [92].
New instances on nodes and relations are set to arrive sequentially.
There are several baseline models used for comparison:
1) join learning as an upper bound, 2) finetuning as a lower
bound that accommodates the new information continuously
without any knowledge consolidation, and 3) general lifelong
learning, EWC [19]. The highest result of the experiment
achieved 94.27 of H @10 (average top-10 ranked entities)
score in the tripe classification task on the WN18 dataset. It is
better than EWC and finetuning methods and almost equal to
the upper bound joint learning technique.
VI. HYBRID APPROACH
The hybrid approach combines more than one lifelong learning
approach to take advantage ofeach approach and maximize the
performance ofmodels [23], [24], [101]. Examples ofthe graph
lifelong learning models that implement hybrid approaches discussed
in this section include ContinualGNN[10], LDANE [72],
and TrafficStream [44].
A. ContinualGNN
In the actual scenario model of ContinualGNN proposed by
Wang et al. [10], there is a concept of replaying strategy to
refine the network as a complement to the regularization
approach itself. It combines both approaches to mitigate catastrophic
forgetting and maintain the existing learned pattern.
The first main goal of this model is to detect a new pattern in
graph structure that significantly influences all nodes in the
network. Second, the model aims to consolidate the knowledge
in the entire network. The ContinualGNN model captures
new patterns in streaming graph data. The model
proposes a method based on a propagation process to efficiently
mine the information on affected nodes when learning
new patterns. The existing knowledge from the propagation
process is then maintained using a combination of both
approaches ofrehearsal and regularization strategies.
During the detection process for new patterns, ContinualGNN
considers the response to the changes by looking at
the number of changes in the networks. It is insufficient to
consider new patterns when the changes are too small, for
example, by only adding an edge in the network while the
other neighborhood in the graph is very stable. Retraining
that scenario is costly and a waste of time. When nodes'
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
