IEEE Computational Intelligence Magazine - February 2023 - 45

representation keeps evolving over time, the model will consider
retraining the network with some strategies to make it
efficient. A scoring function to calculate the influenced degree
ofa node that corresponds to the new patterns defined as:
IðDGtÞ¼fu jk Dht;L
u k¼k htþDt;L
u
ht;L
u k> dg:
u
(9)
A higher score reflects that a node has a high influence. To
calculate that new node representations, htþDt;L
with the previous representation version ht;L
is compared
u
and results the
influence scores. The model also uses threshold values d to
consider the number ofnodes treated as a new pattern. Therefore,
smaller threshold values will consider more nodes as new
patterns only when the result of the influence score is above
the thresholds and vice versa. The model limits to calculating
the scoring function for all nodes at each time step because not
all nodes are affected by the changes. Instead, the model only
calculates it within the order L neighbors that are affected.
Furthermore, the model proposed two prospectives of dataview
and model view, to consolidate the existing patterns.
Data-view aims to maintain memory stability and select only
important information of nodes and neighbors of some graph
data to be saved and revisited during incremental training.
ContinualGNN proposes a step-wise sampling strategy based
on a reservoir sampling algorithm to achieve that objective
[102]. Model view, on the contrary, aims to solve the
overfitting phenomenon by replaying a small amount of data
in memory. This process is highly correlated to the regularization-based
approach by using a generalization of knowledge
preservation derived from a regular lifelong learning algorithm,
EWC [19]. This strategy maintains the distance between the
current model parameters and the previous model parameters
so that it helps to maintain the knowledge and performance of
prior tasks in GNN.
Node classification case study is used to calculate the method's
effectiveness. Some datasets, such as Cora [75], Elliptic [103],
and DBLP [88] are used. Some baselines can be used, such as
GNN with a retraining process as the upper bound and GNNs
with incremental scenarios like finetuning as the lower bound.
ContinualGNN achieves a maximum average accuracy of
0.9212 on the elliptic dataset. It proves to have better performance
than the lower bound ofGNNs with a simple incremental
learning scenario and has almost similar performance to the
upper bound methods ofGNNs with ajoint training strategy.
B. Lifelong DynamicAttributed Network Embedding
Wei et al. [72] developed LDANE that aims to represent
learning to produce a low-dimensional vector for each node
in a growing size network. LDANE is developed based on lifelong
learning that uses an architectural approach, namely
Dynamic Expandable Networks (DEN) [25]. DEN can
dynamically determine network capacity to learn and share the
structure among different tasks. Moreover, LDANE constructs
attribute constraints that can be updated efficiently when the
node attribute changes to restrict the learned embedding to
Lglob ¼
Xn
i¼1
kð^Si SiÞ bi k2
2 :
(10)
the weighted adjacency matrix of graph G is denoted by S.if
there is a relation from node vi and vj, then Sij > 0; otherwise
Sij ¼ 0.
S^i SiÞ defines a new reconstruction value of adjacency
matrix and represents Hadamard product, bi denotes
as a vector with bij ¼ 1if Sij ¼ 0; otherwise bij ¼ d > 1. It is
not enough to preserve global structure proximity; this model
also calculates Lloc to preserve the local structure ofembedding
ofits neighbor node vj that is defined as follows:
Lloc ¼
Xn
i¼1
Sij kðyi yjÞk2
2;
(11)
where yi and yj denotes the result of node representation
between observed node vi and vj. LDANE also constructs two
node sets posi that contains top l similar nodes vp 2 posi and
negi that contains top l dissimilar nodes vm 2 negi. The third
loss function, Lattr forces embedding to satisfy the constraints
based on the top similar and dissimilar nodes:
Lattr ¼
X
vi2V
X X
vm2negi vp2posi
maxð0; simðyi; ymÞ simðyi ypÞÞ (12Þ
where simðyi; ymÞ calculates cosine similarity between two
embeddings of node vi and a dissimilar node vm. simðyi ypÞ
calculate cosine similarity between two embeddings ofvi and a
similar node vp. The main purpose of that loss function is to
ensure that the similarity ofsimilar node embeddings is not less
than the similarity of dissimilar node embeddings. LDANE
combines all objective functions of global structure proximity,
local structure proximity, and attribute similarity together as
follows:
L ¼ Lglob þ a1Lloc þ a2Lattr þ v1L1 þ v2L2;
(13)
where a1, a2, v1, v2 are hyperparameters for relative weights of
objective functions. L1 and L2 are regularizers to prevent the
network weights from overfitting.
LDANE performs several processes, such as output expansion,
selective retraining, autoencoder expansion, and autoencoder
split, to enable lifelong learning on the network.
Furthermore, it applies all approaches simultaneously. First,
input and output expansion aims to accommodate a dynamic
network that constantly changes. When the number of nodes
at the following time observation is greater than the previous
FEBRUARY 2023 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 45
satisfy the constraints. In this method, a Deep autoencoder is
employed to learn the embedding of nodes. LDANE uses
three kinds of loss functions of Lglob, Lloc, and Lattr. For any
nodes vi, vj, vp and vm, where vi is the observed node, vj is the
neighbor node ofvi, vp is the most similar node-set in terms of
attributes of node vi, and vm is the most dissimilar node set of
attributes of node vi. Lglob calculates the reconstruction error
ofoutput and input or error ofglobal structure approximation
that is shown as follows:
IEEE Computational Intelligence Magazine - February 2023

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2023
