IEEE Computational Intelligence Magazine - May 2021 - 18

to evolve neural network weights w in continuous domains, by
using a d-dimensional multivariate normal search distribution
[37], [42], [46]-[49].
For demonstration purposes, we consider the state-ofthe-art NES as the baseline neuroevolution algorithm in
this paper. The underlying objective function of NES can
be expressed as:

J ^ih = E i 6 f ^w h@ =

f ^w h r ^w|ih dw, (3)

which is the expected fitness of the population represented by
probabilistic model r ^w|i h, where the search distribution
is specified by distributional parameters i. In this paper, the
d-dimensional multivariate normal search distribution is
parameterized by i = ^ n, Ah, where A AT = R. The distribution mean n ! R d and full covariance matrix R ! R d # d
characterize the search center and mutation. A candidate
solution for the neural network weights is therefore a realization of the nor mally distr ibuted random var iable
w + N ^ n, R h .

Initialization

Source
Distribution

Search
Distribution

Transfer?

Mixture
Model
Update
Mixture Model

Update
Search
Distribution

Evaluate
Fitness

Stopping
Criteria
Met?
Y

Sample
PseudoOffspring
Evaluate
Fitness
Project
PseudoOffspring

Optimized
Solution

FIGURE 3 A conceptual illustration of the proposed mixture modelbased adaptive transfer, featuring the methodology (1) to construct
and update a mixture model of the target search and source distributions, and (2) to influence the search distribution update on the target problem based on the projection of pseudo-offspring sampled
from the source distributions (boxes highlighted in red). The proposed transfer method is suitable for general probabilistic modelbased evolution strategies as shown on the left side of the vertical
dotted line, with their interface indicated by dashed arrows.

d i J ^ih . 1
m

/ f ^w kh d
m

k=1

log r ^w k|ih, (4)

where m is the population size and f ^w kh is the fitness of the
kth sampled offspring. Then, the distributional parameters are
updated based on the estimated search gradient as:

! i + h $ F -1 d i J ^ i h, (5)

where h is the learning rate. In (5) the search gradient is normalized by the inverse of the Fisher information matrix
F = E i 6d i log r ^w|i h d i log r ^w|i hT @ (i.e., the variance of
the gradient) for the given distributional parameters to form
the natural gradient P
d i J ^i h = F-1 d i J ^i h, which is the hallmark of NES. This natural gradient takes into consideration the
uncertainty of gradient estimates when providing an ascent
direction in the space of distributional parameters.
IV. Transfer Neuroevolution

N
Sample
PseudoOffspring

It is worth noting that NES searches in the space of distributional parameters, but not the problem space. In contrast, the
objective function for typical gradient descent methods such as
SGD is to search for an optimal w directly; i.e., J ^w h = L ^w h.
This further highlights the conceptual distinction between the
two approaches.
In particular, NES utilizes the search gradient d i J ^i h on the
expected fitness (3) to evolve the search distribution. The gradient can be estimated from a population of pseudo-offspring
drawn from the current search distribution as [37]:

IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2021

In this section, we augment neuroevolution with transfer optimization, boosting search efficiency by reusing experiential
priors from related source problem instances. This capability is
enabled via a mixture model-based adaptive transfer method.
The proposed method is suitable for general probabilistic
model-based evolutionary strategies. It is worth noting that
the approach makes no strict assumption on the synergy
between source and target problems. The adaptive transfer
method features a dynamic mechanism to automatically
exploit useful experience from source problems, such that
better quality pseudo-offspring can be induced to effectively
influence the target search distribution update. Importantly,
the adaptation mechanism is able to retreat from irrelevant
sources, to curb negative transfer. Figure 3 gives the conceptual illustration for the proposed mixture model-based adaptive transfer method.
A. Transfer via Mixture Modelling

Let us consider a search distribution r(w|i) for the target optimization problem, and a single source distribution {^w h for simplicity of exposition. It is assumed that the source is expressed in
the form of a distributional prior, for example, a search distribution acquired from successful evolution of the same neural network with NES (or any other probabilistic model-based
evolution strategies like CMA-ES or OpenAI-ES) for solving a
similar differential equations problem. To facilitate knowledge

IEEE Computational Intelligence Magazine - May 2021

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2021