IEEE Computational Intelligence Magazine - May 2021 - 16

of becoming trapped in local minima could nevertheless hinder
a PINN from approximating the right solution. Although SGD
has been very successful in deep learning, the emphasis is typically to learn a model from noisy data with good out-of-sample generalization. Over-fitting to the training data is
discouraged, and a local minimum often suffices in achieving
high test accuracy. However, the requirement is somewhat different when it comes to solving differential equations, in which
globally optimum model parameters are sought. Generalizability and over-fitting are not a concern given access to potentially
infinite training data covering the entire problem domain; rather, having a pre-maturely converged physics-informed neural
network implies an unphysical solution. For these reasons, it is
contended that SGD may not necessarily be the best approach
for optimization in this domain.
An alternative approach comes from the field of neuroevolution [33], which uses evolutionary algorithms (EAs) to optimize
PINNs. Their main conceptual distinction from SGD is that
SGD follows a single gradient direction, whereas EAs search
with a population of diverse solutions with the goal of circumventing local optima [34]. This makes neuroevolution a different paradigm from gradient descent, since the notion of
diversity does not explicitly exist in the latter [35]. As such,
neuroevolution offers a promising substitute to SGD for global
optimization [36]; such as for solving differential equations. In
particular, it is demonstrated in this paper that neuroevolution
via a state-of-the-art EA, namely natural evolution strategies
(NES) [37], outperforms SGD in solving a variety of differential equations, providing physically accurate solutions. Neuroevolution is thus highlighted as a noteworthy approach for
solving differential equations.
Neuroevolution can however be slow to converge compared to gradient descent. To enhance computational tractability, we propose a novel augmentation of neuroevolution
with transfer optimization [38]-[40], where information in the
form of experiential priors are reused from past (source) problem
instances to boost the target search. In a scientific study, it is
common for a single set of differential equations to be evaluated under different environments and boundary conditions.
Relevant experiences are therefore naturally accumulated
over time, and can be exploited given any new target problem. To this end, we develop a new transfer optimization
method which can be integrated with probabilistic modelbased evolution strategies, such as NES (or even others like
CMA-ES or OpenAI-ES [41], [42]). Different from the commonly used transfer strategy of fixing pre-trained neural network layers [43], [44], our method features a probability
mixture model-based adaptive design to protect the algorithm
from the risk of negative transfer. The mixture model allows
the joint processing of multiple sources, adaptively selecting
the one that is most relevant to the target task. In the experimental study, the proposed transfer optimization algorithm is
integrated with the NES, empirically demonstrating improvements in convergence speed and accuracy over baseline neuroevolution and SGD.

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021

The remainder of the paper is organized as follows. In the
next section, physics-informed neural networks for differential
equation problem is described. In Section III, the methodology
of neuroevolution via probabilistic model-based evolution
strategies is introduced. The proposed method for achieving
transfer neuroevolution is then presented in Section IV. Section V presents the experimental study to illustrate the competitive advantage of transfer neuroevolution over SGD across
several test problems. Finally, Section VI contains the conclusion
and directions for future research.
II. Problem Setup: Physics-Informed Neural
Networks for Differential Equations
A. Differential Equations

Consider differential equations of the general form:

u t ^x, t h + N x 6u ^x, t h@ = 0, x e X, t e 60, T @, (1)

u ^x, 0h = u o ^ x h, x e X, (1b)

B 6u ^x, t h@ = g ^x, t h, x e 2X, t e 60, T @, (1c)

where u t ^x, t h is the temporal derivative, and N x 6 $ @ is a general nonlinear differential operator which includes non-linear
terms of spatial derivatives, such as the first and second order
derivatives u x ^x, t h and u xx ^x, t h, respectively. The differential
equation (1) usually describes certain dynamical processes in
the physical world. The corresponding spatial domain can be of
1-, 2- or 3-dimensions. As an example, the Burgers' equation
reads u t + u $ u x - o $ u xx = 0, by having N x 6u@ = u $ u x o $ u xx . The nonlinear differential operator in Burgers' equation describes the advection and diffusion processes of the
physical quantity u in 1 D, and contains a problem specific diffusion coefficient o.
The interest of this paper lies in finding the solution u ^x, t h
which satisfies the differential equation (1) across space
x e X and time t e 60, T @ domains. Such a solution may not
be unique unless sufficient initial condition (1b) and/or
boundary condition (1c) are given. The initial condition at
t = 0 is defined by u o ^ x h, and the boundary operator B 6 $ @
enforces the desired condition g ^x, t h at the domain boundary 2X. This B 6 $ @ can be an identity operator (Dirichlet
boundary condition) or a differential operator (Neumann
boundary condition). A meaningful solution would need to
satisfy all these conditions, in addition to the main differential equation.
B. Physics-Informed Neural Networks

To solve the differential equation in (1), the neural network
approach constructs a PINN representation ut ^x, t; w h to
emulate the unknown solution u, with network parameters
w = " w i ,di = 1 to be optimized. In the present study, the network architecture and other hyper-parameters are specified, making w refer to the weights of the neural network.
Such a neural network is termed as " physics-informed
neural network, " because it uses the residual terms from

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021