IEEE Computational Intelligence Magazine - February 2023 - 43

In graph lifelong learning, consolidating
knowledge is necessary to alleviate
forgetting problems pertaining to
changes in the graph representation in
terms of nodes, relations, and tasks.
performance of the method. First, as a lower bound, finetuning
is implemented to train the data continuously without
employing lifelong learning. Second, as the upper bound, all
nodes are retrained from scratch when new information
appears. Moreover, there are some general lifelong learning
that does not consider graph representation learning, such as
EWC [19], EMR [16], and GEM [23]. For link prediction,
DiCGRL achieves the maximum average performance ofH@
10 (average top-10 ranked entities) on the FB15K-237 dataset
with 47.7%. The highest average accuracy on the Pubmed
dataset for node classification is 85%. DiCGRL achieves an
improvement compared to the general lifelong learning techniques
and tends to have a close result with the upper bound
baselines.
B. Graph Few-Shot Class-Incremental Learning
The ability to learn new classes incrementally is essential in
real-world scenarios. Zhen et al. [69] proposed GPIL, which
implements lifelong learning in graphs to learn both newly
encountered and previous classes. To enable a lifelong
learning setting, firstly, the model spans the dataset into two
parts, i.e., base classes and pseudo novel classes, with disjoint
label spaces. Then, Graph FGL recurrently pre-trains the
encoderonthe base classesand keepsitduringthe pseudo
incremental learning process. Then it performs meta-learning
to learn initialization with more transferable metaknowledge
with a novel pseudo incremental learning,
GPIL. During each meta-training episode, All few-shot
classification tasks are sampled from pseudo-novel classes
and base classes to imitate the incremental process in the
evaluation process. By doing that, each meta-learning episode
is able to learn a transferable model initialization for
the incremental learning phase. In order to minimize the
catastrophic forgetting problem, this method proposes a
Hierarchical Attention Graph Meta-learning framework
(HAG-Meta) by using the regularization technique to modify
the loss function during the learning process. It uses a
dynamically scaled loss regularizer that utilises the scale factor
[95], [96] to multiple task levels to adjust their contribution
to model training. That scaling factor helps to reduce
the contribution of insignificant classes while maximizing
the important tasks. Moreover, This method proposed two
hierarchical attention modules: 1) tasks-level attention,
which estimates the importance of each task to balance the
contribution of different tasks, and 2) node-level attention,
which maintains a better balance between prior knowledge
and new knowledge within the nodes. The model claims to
have excellent stability in consolidating knowledge and
gaining advantageous adaptability to new knowledge with
minimal data samples.
The experiment conducted on GPIL uses a few-shot learning
case study with graphs offew-shot learning datasets such as
DBLP [88], Amazon-clothing [97], and Reddit [35]. It uses a
split strategy for the class into 3 parts pre-training, meta-training,
and evaluation. GPIL is compared with several methods
such as continual learning on graphs, ER-GNN [9], standard
continual learning ICARL [51], which uses modifications
with GNN, Few-shot Class-incremental Learning (CEC) [98],
etc. The experiment calculates the performance dropping rate
(PD) that measures the accuracy drop from the last sessions.
There are 10 sessions in every experiment of each baseline
method. GPIL achieve the lowest performance dropping rate
(PD) in the DBLP dataset with 17.11%. It is better than other
baseline methods in terms of minimizing performance
degradation.
C. Topology-Aware Weight Preserving
The TWP method by Liu et al. [70] proposed a novel
method to strengthen lifelong learning and minimize catastrophic
forgetting in GNNs. TWP explicitly studies the
local structure of the input graph and stabilizes the important
parameters in topological aggregation. Given an input
graph and its embedding feature of nodes, TWP estimates
the important score of each network parameter based on
their contribution to the topological structure and taskrelated
performance. The methods used in TWP are to calculate
the gradients of the task-wise objective and topological
preserving with each parameter, then consider the
gradient as an index for the parameter importance. After
learning previous tasks, the model gets the optimal parameters
by minimizing the loss on the task. As not all parameters
contribute equally, it is important to preserve the
minimized loss by considering highly influential parameters.
Approximate contributions of each parameter are calculated
basedonaninfinitesimal change in each parameter, like in
the general lifelong learning method of Synaptic Intelligence
(SI) [99]. Parameters that significantly contribute to
minimizing loss must be kept stable when learning future
tasks. Besides minimized loss preserving, topological structure
preservation is conducted since structure information in
the graph plays an important role. It aims to find the parameterswitha
substantial contributiontolearningthe topological
information of the graph. GAT [78] are employed to
learn topological information around the center node by
calculating the attention coefficient of its neighbor. With
the graph modeling process by GAT, then, the infinitesimal
changes of each embedding feature are calculated to get the
importance scores of each parameter. The regularization
approach accommodates the new task by minimizing the
performance degradation of the previous task. TWP penalizes
changes in important parameters for old knowledge by
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 43
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
