IEEE Computational Intelligence Magazine - May 2021 - 15

novel and computationally efficient transfer neuroevolution algorithm is proposed in this paper. Our method is capable of
exploiting relevant experiential priors when solving a new
problem, with adaptation to protect against the risk of negative
transfer. The algorithm is applied on a variety of differential
equations to empirically demonstrate that transfer neuroevolution can indeed achieve better accuracy and faster convergence
than SGD. The experimental outcomes thus establish transfer
neuroevolution as a noteworthy approach for solving differential equations, one that has never been studied in the past. Our
work expands the resource of available algorithms for optimizing physics-informed neural networks.

I. Introduction

Meshing (i.e., spatial discretization) itself is a nontrivial task.
Classical numerical schemes are prone to failure in finding the
right solution if the meshing is not appropriately done. Their
accuracy is limited by the size of the discretization and interpolation scheme used (usually linear). Being universal approximators, PINNs offer superior approximation or even exact
replication to the solution. Moreover, the solution via PINN is
differentiable. Its relatively compact representation requires
lower memory demand for storage. Beyond solving differential
equations (referred to as the forward problem), physicsinformed neural networks can be further extended to solve
inverse problems such as designing metamaterials [26], inferring
unknown dynamic from observations [27], [28], and quantifying
fluid flows from visualizations or sensors data [29]-[31]; see
examples in Figure 1. They offer a path to rationalizable artificial
and computational intelligence systems that are consistent with
fundamental physics laws [32]. These advantages brought by
synergizing PINNs and differential equations are however
attainable at the cost of a steep optimization challenge caused by
high-dimensional, non-convex loss function landscapes.
When considering the optimization of physics-informed
neural networks, there has been a lack of research investigation beyond mainstream gradient descent methods, such as
stochastic gradient descent (SGD) and Broyden-Fletcher-
Goldfarb-Shanno (BFGS) algorithms. Their known tendency

olving ordinary differential equations (ODEs) and partial differential equations (PDEs) is the cornerstone of
scientific modelling in modern science and engineering
[1]-[3]. The solution of these differential equations allows
us to understand and make predictions on the behaviors of
physical systems in a wide variety of scientific problems,
including heat and mass transfer, optics, acoustics, material elasticity, electromagnetic waves, fluid dynamics, and many other
dynamical processes in physics [4], [5], sociology [6], finance
and economics [7]. The idea of using neural networks to solve
differential equations goes back to early works in the 1990s
[8]-[14]. Given the surging popularity
of deep learning lately and advancement in computing capability, there has
been renewed interest in designing
Inverse Problem
Forward Problem
deep neural networks (DNNs) to solve
Solving Differential
differential equations [15]-[23]. In brief,
Equations
a DNN-also termed herein as a physics-informed neural network (PINN) as
per Raissi et al. [24]-is constructed
such that its output ut ^x, t h generates
x
Metamaterials
Study Quantum
the solution u of the governing differDesign
Mechanical
NN
u
System
ential equations in space x e X and
t
time t e 60, T @ domains. The loss function of such a PINN is defined by the
Physics-Informed
residual terms from the differential
Neural Networks
equations, under prescribed initial and
boundary conditions. In principle, infinite training data instances are accessible
Quantifying
Hemodynamics in
Convection/Heat
by sampling arbitrary points within a
the Vasculature
Transfer Process
problem's spatial-temporal domain. The
loss thus acts as a penalty to constrain
the PINN from violating the governing equations at all points. In other words,
the problem of solving differential equations
is transformed to one of global optimization,
in which the PINN loss is to be minimized
Aerodynamic
Inferring Unknown Fluid Dynamics
to zero.
Different from classical numerical
FIGURE 1 Applications of physics-informed neural networks. By solving differential equations,
solvers [25], the PINN approach has the accurate predictions on the behaviors of physical systems can be made in a wide variety of probmain advantage of being mesh-free. lems across scientific fields. PINNs can also be extended to solve inverse problems.
1

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021