IEEE Computational Intelligence Magazine - May 2022 - 16

In our work, a multi-head feedforward neural network
component for predicting values of properties is
added to the original model such that the latent
representations can be regularized by properties
of interest.
using graphs as molecular representations in VAE, a technical
difficulty is to design an effective graph decoder. In addition to
other heuristic methods, a SMILES decoder can be used to
pair with a graph encoder. In recent work, 3D conformation
representation, modeling, and generation show promising
results for solving some of these technical challenges for
molecular design [16].
While DGMs offer the convenience of searching in latent
space, RL algorithms can directly search in the molecules'
structural space by adding or deleting atoms and bonds [17]. In
the Markov decision process (MDP) for drug design, the agent
is a molecular generator, the molecular structure indicates the
state, the actions are modifications to the current structure, and
a simulator (e.g., surrogate model) is often used as the environment
to provide rewards. Furthermore, generative and predictive
models can be integrated in MDP to form deep
reinforcement learning (DRL) methods, where the generative
model is trained as a policy approximation and the predictive
model can be used as a value function approximation [18],
[19]. Search in the discrete input space and inefficient learning
are arguable concerns to be addressed when applying RLbased
solutions for compound design.
Interestingly, as an old peer of RL for black-box optimization,
EC methods [20] also achieve promising performances in
modern optimization, design, and modeling problems. In [21], a
strategy is presented for evolution of ligands along multiple
properties in the structural space, where a library of knowledge-based
chemical structural transformations is used as the
mutation operator. Interactions between EC and neural networks
have mainly focused on network evolution and neural
surrogate models for fitness functions. For example, EC has
been used at a large scale for neuroevolution that leads to evolution
of neural network architectures [22]; feedforward neural
network can also be used as a fitness function in EC [23].
Moreover, it has been recently discovered that evolutionary
strategy (ES) can perform competitively with RL in game AI
[24]. ES [25] and estimation of distribution algorithms (EDA)
[26] build parameterized search distributions over promising
points and either employ gradient information or sample from
such a probabilistic model to find better points. Both probabilistic
strategies from EC can potentially be used as alternatives
to Bayesian optimization (BO) in (continuous) black-box
optimization. A single-objective BO has been recently applied
to molecule optimization in the latent space of VAE [10].
Since model learning is essentially parameter estimation
from the statistical modeling perspective, the quality of data in
16 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | MAY 2022
III. Method
The proposed deep evolutionary learning (DEL) process combines
deep learning and multi-objective EC through the latent
representation space of molecules. One of the theoretical innovations
of our approach is that it demonstrates that EC methods
are extendable to corresponding deep versions. The main
idea is illustrated as a flowchart in Figure 1 and formally presented
in Algorithm 1 (see Section S.1 in the Supplementary
Materials). This algorithm consists of the following steps. (a) A
VAE (as molecule modeler) and a multilayer perceptron neural
network (MLP, as property predictor to regularize the latent
space) are pretrained using all the original training data to initiate
the first evolutionary generation, or are fine-tuned in
subsequent generations using samples from the previous population.
(b) Training samples (if first generation) or population
samples (otherwise) are projected to the latent space using the
encoder of the VAE. The latent vectors are used as individuals
for subsequent evolutionary operations. (c) Based on nondominated
ranking and crowding distances of samples with
respect to multiple properties (or other multi-objective sorting
methods), evolutionary operations (selection, recombination/
crossover, and mutation) are conducted on the latent representations
of the samples. (d) Given these new latent codes after
evolutionary operations, new molecule samples are generated
by the decoder of the VAE. (e) Properties of these generated
samples are obtained using a simulator (e.g., RDKit [32] in
our experiments). (f) New samples with good desired properties
and good samples from the previous generation form the
new population. (g) Steps (a-f) are iterated for multiple generations.
(h) The final population is returned. Please note, RDKit
is used as a simulator to calculate properties of new molecules.
The property predictor is merely designed to regularize the
generative model.
The major advantages of our DEL algorithm over existing
interactions between EC and neural networks can be explained
as follows. (a) DEL directly evolves a collection of data rather
than many neural network structures and parameters. Data evolution
tends to be more efficient than direct evolution of
deep learning is crucial for model performance.
Data augmentation is becoming a new
strategy in deep learning to improve the training
of a model. For example, in computer
vision, transformations (such as rotation and
flipping) of images are used to increase the
sample size when the original dataset is insufficient
[27]-[29]. In NLP, a text dataset can be
augmented using tricks such as replacing
words or phrases with their synonyms [30], and resorting aids
from other language models (e.g., word embedding and neural
machine translations) [30], [31]. Basically, these methods either
increase the data by transforming existing information which
can only alleviate the limit of certain techniques (e.g., convolution),
or indirectly borrow new information from other sources
(e.g., methods in NLP).
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
