IEEE Robotics & Automation Magazine - June 2020 - 27

parameter query are almost equal and close to 1/K, each mode
has the same probability of being selected to generate motions.
Hence, the mode collapse does not occur. In the latter case, if
each model is associated with some demonstrations, the corresponding mixture component, i.e., the network branch, is well
trained. Hence, model collapse does not occur.
The objective function for training the entropy MDN is a
weighted sum of the NLL and the entropy cost function:
w NLL l NLL + w model l model . In the following experiments, the
weights are empirically determined: w NLL = 1, w model = 50.
Improving the MDN With Failures
In many applications, the failed samples are easily collected
with an underfitted MDN model. To reduce the occurrence of
these failed MP parameters for similar task parameter queries,
we introduce the failure cost function as follows:
l neg (H) =

i=1

k=1

/ log e / r k(q i; H) N (w i); n i, R neg) o,

(9)

where the normal distribution has the same form as (6), but
R neg = v neg I. By minimizing this cost, the output mean vecq i is kept away from the
tor n i for a specific task parameter
)
) M
"
,
w
.
failed MP parameters i i = 1
If v neg is too small, the failure cost does not affect the
results; on the other hand, a v neg that is too large can result in
trajectories that are significantly different from demonstrations. Here, we determined v neg empirically with the smallest
variance of all MP parameter components.
To train an MDN with the failure cost, we prepare an evaluation data set. After a certain number of training steps, we
run the MDN on this evaluation data set and collect the failed
)
samples in a failures data set {w )i } iM= 1. In the next training
steps, we calculate the failure cost function based on the failures data set. To avoid increasing the computational cost, we
use a fixed data set size M ) and remove the earliest failed
samples when new samples are collected.
To evaluate the MP generalization methods, we check
whether the generated MPs accomplish the tasks with different task parameters. In learning from demonstrations, a successful task execution means that the generated motions are
similar to the demonstrations and can accomplish the task
with specific task parameters. In the proposed method, we
meet this requirement by training the MDN with the NLL,
which is related to the similarity between the collected and
generated MP parameters. To check whether the trained
model meets the latter requirement, we evaluate it with the
success rate of task execution.
For task execution, we can only execute one motion after
another. Hence, the MP parameter for the task must be determined based on the MDN output distribution in a subsequent step.
Generating Motion With the MDN
The purpose is to generate single motions for some task
parameter queries. In the following experiments, we consider
two strategies: selecting the most probable mode or choosing

the best one from multiple samples. The most probable mode
is the output mean vector of the mixture component that has
the most significant mixing coefficient. If the Gaussian components of the output GMM have separated means, the most
probable mode corresponds to the mode of the GMM.
For one specific task parameter query q, the MDN outputs
K Gaussian mixture components with their mixing coefficients " r k ,Kk = 1 . The K modes of these Gaussian mixture components correspond to the K most probable motions of
different types. However, not all of these K modes can accomplish the task. The most significant mixing coefficient indicates the mode that most likely succeeds. With this strategy,
the MDN serves as a deterministic model; hence, we can
compare it with previous deterministic methods. Selecting the
most probable mode is the simplest way to generate motion
from the MDN output. Moreover, this strategy works quite
well in many tasks.
However, with the most probable mode, we ignore the
information provided by the output variance R of the
MDN. Each of its diagonal elements indicates how various
the generated trajectories can be at the corresponding time
for successful task execution. When we draw samples from
the output GMM for a specific task parameter query, the
variance matrix ensures that the samples have a high probability of success. In some tasks, the most probable mode
does not work very well, such as in the experiment
described in the "The Hit-the-Ball Experiment in MuJoCo" section. To improve the performance, we draw several
MP parameters from the output distribution and execute
one after another for the task until success. In this case, the
success rate is also dependent on the number of samples.
Moreover, with the former strategy, the MDN always generates the same motion for one specific task parameter query.
To demonstrate motion diversity and the fact that the MDN
learns multiple modes, we also must draw multiple samples
from the output distribution (see the "The Throw-the-Ball
Experiment" section).
In the next section, we evaluate the proposed method with
four experiments. In the first two investigations, we select the
most probable mode and compare our method with previous
deterministic methods. In the other two, we draw multiple
samples from the MDN output to either improve the performance or show the motion diversity.
Experiments and Evaluations
Fitting Polynomials
In this experiment, we consider a fifth-order polynomial
y (x) = / 5k = 1 a k x k. The purpose is to learn a mapping from
the coefficients a k to the MP parameter w in (2). The error is
the distance between the true fifth-order polynomials and the
generated trajectories by the output VMP parameters. We
evaluate different methods separately for the inputs with one
dimension a 5 to all dimensions (a 5, a 4, f, a 0)T. Figure 4
shows the results for 60 experiments. In each experiment,
30 random coefficients are for training, and 20 are for testing.
JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020