IEEE Computational Intelligence Magazine - May 2021 - 22

2 boundary points are matched against the
boundary conditions. We refer to such computation as a single evaluation of the neural network loss. Since tNES, xNES and SGD
algorithms are all stochastic in nature, multiple
optimization runs are performed for each
problem to statistically assess their results.
The results are summarized in Figure 5. At
larger v value, the solution u ^ x h is characterized by an extended flat region followed by
steep gradient curvature near the end of x domain (Figure 5a).
As the nonlinearity of the solution increases with velocity v, the
underlying differential equation is more difficult to match. This
makes the optimization problem become increasingly challenging. As a consequence, some of the solutions returned by SGD
optimization are not meaningful, because they could not even
fulfill the boundary conditions (Figure 5b). These poorly optimized solutions have relatively high residual terms from the
boundary conditions at two ends of x, and from the differential
equation over the entire domain. In contrast, the worst solution
from neuroevolution performs much better than SGD in terms
of the optimized loss (aggregated residual from the differential
equation and boundary conditions) and the error against
ground truth solution. For example, at v = 8, the aggregated
residuals from the worst xNES and SGD solution are 1.3e-5 vs.
4.1e-1, and their mean squared errors against ground truth are
2.3e-7 vs. 2.3e-1.

By altering the neural network weights, the differential
equation as well as the prescribed initial and/or
boundary conditions must be satisfied. The resultant
global optimization problem can often be very
challenging even if the differential equation looks
simple on the surface.
to failure at high velocity values caused by inappropriate
-meshing, which gives rise to unphysical oscillations in the solution [52]. Being mesh-free, the PINN approach avoids this issue.
A physics-informed neural network with 45 tunable
weights is constructed to emulate the solution u given input x.
Specifically, the loss function for the 1D steady state convection-diffusion equation (12) is defined as follows:

	

L = L DE + L BC
m
= 1 / ^k $ ut xx - v $ ut xh2 + 6ut ^ 0 h - 0 + ut ^L h - 1@ . (14)
m i=1

Note that in this example, the loss function (14) computes the
mean squared residual of the differential equation over
m = 1000 + 2 collocation points, i.e., 1000 randomly sampled
points within domain x e 60, 5@ using Latin hypercube sampling, plus the 2 boundary points x = 0, x = L . In addition, the

TABLE II The physic-informed neural network and optimization configurations used in experimental study.
A

B

C

1D steady state
-convection-diffusion

2D projectile motion

Model equations of traveling waves

Example

(1) Linearized Burgers
(2) Nonlinear Burgers

(3) Korteweg-de Vries
(KdV)

PINN architecture

^ x h - 5 - 5 - ^ut h

^ t h - 3 - 3 - 3 - 3 - ^ xt , yt h

^ x, t h - 4 - 4 - 4 - ^ut h

^ x, t h - 4 - 4 - 4 - 4 - ^ut h

no. collocation points m sampled for
1 -evaluation of loss L PDE + L BC/IC

1000 + 2

1000 + 1

5000 + 50

10000 + 100

max. evaluation

2e5

2e5

2e5

3e5

tNES
&
xNES
-setting

population size m

20

20

20

30

learning rate
lr = " h d, h M, h a ,

1, 5e-2, 5e-2

1, 5e-2, 5e-2

1, 1e-2, 1e-2

1, 1e-2, 1e-2

initial search distribution
" n, vI ,

0, 5e-2

0, 5e-2

0, 5e-2

0, 5e-2

tNES transfer plan: " Tt, t max ,

2, 500

2, 500

2, 500

2, 1000

SGD
(ADAM)
setting

max. evaluation

2e5

2e5

2e5

3e5

initial learning rate

5e-2

5e-2

1e-2

1e-2

learning plan

reduce learning rate by half on plateaus, with a min. learning rate set at 1e-6

(1) For the PINN architecture, the numbers in between input and output represent the number of nodes in hidden layers. For example, (t) - 3 - 3 - 3 - 3 - ( tx, yt ) indicates a
neural network with single input t, followed by 4 hidden layers with 3 nodes in each layer, and multi-output ( tx, yt ) . All hidden layers, except the final hidden layer, include a bias
term and use 'tanh' activation function. The final hidden layer uses 'linear' activation function and does not include a bias term.
(2) In tNES, r = 12 is used as the Mahalanobis distance threshold for pseudo-offspring projection.
(3) For neural network optimization, in additional to the maximum number of evaluations, a target loss = 1e-9 is also set as stopping criteria.
(4) Keras package is used to perform ADAM. For ADAM parameters except those stated above, default values are used.

22

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021
IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021
