IEEE Computational Intelligence Magazine - February 2023 - 48

TABLE IV Datasets.
DATASET
Citeseer
Cora
OGBN-Arxiv
NODES
3,312
2,708
169,343
EDGES FEATURES TASKS
4,732
5,429
1,166,243
OGBN-Product 2,449,029 61,859,140
OGBN-Proteins 132,534 39,561,252
Pubmed
Reddit
19,717
44,338
232,965 11,606,919
3,703
1,433
128
100
8
500
602
6
7
AT ¼
40
47
112
3
41
A. Datasets
There are no datasets that are specifically intended to check the
performance of graph lifelong learning. Most of the benchmark
datasets used in general lifelong learning are in the form
of image classification tasks, such as Permuted MNIST [19]
and Split CIFAR [23]. Those datasets can still be used as graph
representations, such as changing " superpixels " to represent
nodes in the graph [57]. However, assuming superpixels to
represent nodes in the graph is not close to the real scenario of
graph structure that forms an irregular structure that differs
from a grid structure in the image.
Several popular graph learning benchmark datasets such as
citation datasets (Citeseer [75], Cora [75], OGBN-Arxiv [76],
[77], and Pubmed [75]), User comment datasets (Reddit [35]),
product co-purchasing network (OGBN-product [111]), and
protein interaction network (OGBN-Proteins [112]) can be
used to test the performance of the graph lifelong learning
through scenario modification. The modification ofnew instances
and new classes scenario is by splitting the class or label ofthe
tasks to be distributed at several different times. For example,
there are 10 prediction classes needed to enable an experiment
on graph lifelong learning, those classes need to be split into several
incremental learning scenarios, for instance, five tasks, and it
will result in two classes per task. The process starts with learning
the first task and incrementally learning the rest of the tasks by
sequentially adding new instances and new classes. This experimental
design aims to determine whether performance methods
can perform incremental learning, accommodate new tasks, and
prevent catastrophic forgetting ofprevious tasks. The characteristics
ofthose mentioned datasets are described in Table IV.
B. Performance Metrics
Specifying evaluation metrics before measuring the performance
ofgraph lifelong learning methods is important. Evaluation
metrics in lifelong learning graphs are not similar to
calculating the accuracy oftasks in general machine learning or
graph learning. Chaudhry et al. [12] explained the most common
metric to measure performance in general lifelong learning,
such as average accuracy (A) and forgetting measure (F).
Another proposed work by Chaudhry et al. [113] described a
new measure called the learning curve area (LCA) that calculates
how fast a model learns. The details ofthose three metrics
are as follow:
48 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2023
❏ Average Accuracy (A 2½0; 1) [12]. The average accuracy
of all tasks after processing incremental learning from
the first task to the T-th task is defined as:
1
T
XT
i¼1
aT;i
(16)
where aT;i 2½0; 1 is the accuracy ofindividual task on
i-th task (i T) that calculated after learning incrementally
from task 1 to task T.
❏ Forgetting Measure (F 2½1; 1) [12]. The average forgetting
measure of all tasks after incremental process learning
from the first task to the T task is defined as:
1
FT ¼
where fj
T 1
XT
i¼1
fT
i
(17)
i is the forgetting value on individual task ti
after the model learn up to task tj and it is defined as:
fj
i ¼ max
k 1;...;j1fg
ak;i aj;i
(18)
❏ Learning Curve Area (LCA 2½0; 1) [113]. LCA is the
area underZb curve that calculates average b-shot performance
(where b denotes a mini-batch number). Zb is the accuracy of
each task after observing b-th mini-batch. It will be high ifthe
zero-shot performance is good and a model learns quickly:
1
Zb ¼
T
XT
i¼1
ai;b;i
(19)
IX. OPEN ISSUES
The biggest challenge in lifelong learning settings is catastrophic
forgetting [17] or called catastrophic interference
[114], where a model cannot maintain performance on
previous tasks when learning new tasks, as shown in Fig. 1. In
graph lifelong learning, consolidating knowledge is necessary
to minimize forgetting problems based on changes in the graph
representation in terms ofnodes, relations, and tasks. Examples
of classical approaches that cannot optimally solve catastrophic
forgetting in graph lifelong learning include finetuning and
joint training [47]. Finetuning uses a model from previous tasks
to optimize the task's parameters being learned. When the
model is not guided to learn new tasks, it results in performance
degradation for completing previous tasks. Joint training
is an approach that uses data from previous tasks and new
tasks to conduct training together. It may provide optimal
results but requires all the data that might have been lost,
resulting in process inefficiency when having massive data so
that learning iteratively with a small amount ofdata is needed.
In addition to catastrophic forgetting, there are some other
open issues and challenges to implementing graph lifelong
learning, as summarized in the following points:
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
