IEEE Computational Intelligence Magazine - May 2021 - 17

the differential equation  (1), and the prescribed initial (1b)
and boundary (1c) conditions as the loss function, which is
defined below:
L = L DE + b IC $ L IC + b BC $ L BC, (2)

where,
, (2b)

, (2c)
.(2d)

Here, h ^ y h Y = #Y h ^ y h 2 dy where y e Y. The defined integral loss (2) has a global minimum of zero, given that the differential equation and the initial and boundary conditions are
exactly satisfied everywhere in the problem domain X # 60, T @ .
At L = 0, the PINN output exactly emulates the unknown true
solution of the differential equation. In this regard, we are faced
with a global optimization problem (instead of a learning generalization problem) for finding the most physically accurate solution to the target differential equation.
The relative weights bs in (2) control the trade-off between
different terms in the loss function during the optimization
process, and may need to be scaled in a problem-specific way.
PINNs may also include conventional data loss (for example,
when solving an inverse problem). In the case of solving ODEs
and PDEs, the loss function only comprises of the residual
terms with respect to the differential equation, initial and
boundary conditions; see illustration in Figure 2.
2

C. Computation of the Loss

e.g., ut + u . ux = v . uxx

ux

Initial Condition

e.g., u (x, 0) = -sin(πx)

Boundary Condition

e.g., u(1, t ) = u (-1, t ) = 0

Labelled Data

e.g., u (x, t ) = u

"

t
Physics-Informed
Neural Networks

u

"

NN

uxx
Differential
Operators Via
Automatic
Differentiation

"

PDE Residual

"

ut

"

x

The central thesis of this paper is the untapped efficacy of evolutionary algorithms for optimizing the PINN in generating
good solutions to the differential equations. In particular, the
objective is to find the best d-dimensional neural network
weights w ! R d which minimize the loss (2). As opposed to
typical gradient descent methods such as SGD, which searches
along a single gradient direction, EAs search with a population
of diverse solutions " w k ,mk = 1 for better fitness. Here, m is the
population size and the fitness f = - L refers to the negative
loss of the physics-informed neural network, which is evaluated
on a dynamically sampled batch of m collocation points. The
diversity in an EA population could potentially be the key
-factor to overcome local optima and thus achieve better optimized neural networks.
Among EAs, there is a subgroup of techniques such as NES,
CMA-ES, and OpenAI-ES, that adopt a probabilistic model,
called the search distribution, to represent the population. In a
nutshell, they keep tracing an evolving search distribution and
produce pseudo-offspring by drawing new samples from the
distribution. As the search progresses, the distribution is iteratively updated towards regions with higher population fitness.
All probabilistic model-based evolution strategies follow this
basic principle, although they may differ in their distribution
update mechanism. They have shown to be an effective method

...

The computation of the loss (2) involves substitution of the
PINN output ut into the differential equation for evaluating
the residuals (2b) over the computational domain, as well as
matching the output ut against initial condition (2c) at t = 0,
and boundary conditions (2d) over the domain boundary 2X.
For certain types of activation functions (for example, sigmoid,
softplus, and tanh), the PINN output is higher order differentiable with respect to its spatial and temporal inputs. These differential operators, such as ut t ^x, t; w h, ut x ^x, t; w h, ut xx ^x, t ; w h,

III. Methodology: Neuroevolution

"

2
2X # 60,T @

"
"

L BC = B 6ut ^$ ; w h@ - g ^ $ h

"

	

L IC = ut ^$ , 0; w h - u

2
0 X

2
X # 60, T @

"

	

L DE = ut t ^$ ; w h + N x 6ut ^$ ; w h@

"

	

"

	

are required for the evaluation of the loss. They can be conveniently obtained via automatic differentiation [45].
Although the integral loss terms (2b-d) are defined over a
continuous computational domain, for practical reasons, during
the loss evaluation we compute the mean squared residuals
over a set of m spatial-temporal collocation points
D = " ^x i, t i h,mi = 1. These points are sampled (for example, using
randomized Latin hypercube sampling) from the respective
computational domains. In this paper, a dynamic sampling
scheme is adopted such that for every loss evaluation a new
batch of m collocation points are generated. The evaluated loss
provides us an approximate estimation on how well the differential equation and the prescribed initial and boundary conditions are being satisfied.

FIGURE 2 Physics-informed neural networks consist of additional residual (loss) terms from the differential equation, initial and boundary conditions (blue), which require the computation of differential operators of the neural network output with respect to the inputs. It can emulate the
solution of differential equations without the need for labelled data. We have a global optimization problem (instead of a learning problem) to
find the most physically accurate solution.

MAY 2021 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE

17
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
