IEEE Computational Intelligence Magazine - August 2018 - 68

A generative neural network decodes latent representation
to a data instance, while the discriminative network is
simultaneously taught to discriminate between instances
from the true data distribution and synthesized instances
produced by the generator.
Hu et al. [122] proposed generating
sentences whose attributes are controlled by learning disentangled latent
representations with designated semantics. The authors augmented the latent
code in the VAE with a set of structured
variables, each targeting a salient and
independent semantic feature of sentences. The model incorporated VAE
and attribute discriminators, in which
the VAE component trained the generator to reconstruct real sentences for generating plausible text, while the discriminators forced the generator to produce
attributes coherent with the structured
code. When trained on a large number
of unsupervised sentences and a small
number of labeled sentences, Hu et al.
[122] showed that the model was able to
generate plausible sentences conditioned
on two major attributes of English: tense
and sentiment.
GAN is another class of generative
model composed of two competing
networks. A generative neural network
decodes latent representation to a data
instance, while the discriminative network is simultaneously taught to dis-

z
µ

Linear

LSTM

criminate between instances from the
true data distribution and synthesized
instances produced by the generator.
GAN does not explicitly represent the
true data distribution p (x).
Zhang et al. [121] proposed a framework for employing LSTM and CNN
for adversarial training to generate realistic text. The latent code z was fed to the
LSTM generator at every time step.
CNN acted as a binary sentence classifier
which discriminated between real data
and generated samples. One problem
with applying GAN to text is that the
gradients from the discriminator cannot
properly back-propagate through discrete
variables. In [121], this problem was
solved by making the word prediction at
every time "soft" at the word embedding
space. Yu et al. [123] proposed to bypass
this problem by modeling the generator
as a stochastic policy. The reward signal
came from the GAN discriminator
judged on a complete sequence, and was
passed back to the intermediate stateaction steps using Monte Carlo search.
The evaluation of deep generative
models has been challenging. For text, it

Decoder
y2

LSTM

Encoder
Figure 12 RNN-based vAE network for sentence generation proposed by Bowman et al. [120].

ieee Computational intelligenCe magazine | august 2018

is possible to create oracle training data
from a fixed set of grammars and then
evaluate generative models based on
whether (or how well) the generated
samples agree with the predefined
grammar [124]. Another strategy is to
evaluate BLEU scores of samples on a
large amount of unseen test data. The
ability to generate similar sentences to
unseen real data is considered a measurement of quality [123].
VII. Memory-Augmented Networks

The attention mechanism stores a series
of hidden vectors of the encoder, which
the decoder is allowed to access during
the generation of each token. Here, the
hidden vectors of the encoder can be
seen as entries of the model's "internal
memory". Recently, there has been a
surge of interest in coupling neural networks with a form of memory, which
the model can interact with.
In [135], the authors proposed memory networks for QA tasks. In synthetic
QA, a series of statements (memory
entries) were provided to the model as
potential supporting facts to the question. The model learned to retrieve one
entry at a time from memory based on
the question and previously retrieved
memory. In large-scale realistic QA, a
large set of commonsense knowledge in
the form of (subject, relation, object) triples were used as memory.
Sukhbaatar et al. [136] extended this
work and proposed end-to-end memory networks, where memory entries
were retrieved in a "soft" manner with
attention mechanism, thus enabling endto-end training. Multiple rounds (hops)
of information retrieval from memory
were shown to be essential to good performance and the model was able to
retrieve and reason about several supporting facts to answer a specific question. They also showed a special use of
the model for language modeling,
where each word in the sentence was
seen as a memory entry. With multiple
hops, the model yielded results comparable to deep LSTM models.
Furthermore, dynamic memory networks (DMN) [128] improved upon
previous memory-based models by

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - August 2018