IEEE Robotics & Automation Magazine - June 2020 - 28

The one-step approaches presented in [11] with SVR and
GPR are denoted as "−1," while two-step methods are indicated with "−2." Notice that the dot-product kernel is the perfect
assumption for this task because the polynomial value is,
indeed, a dot product of the coefficients and bases. However,
GPR-1 with dot-product kernels is worse than other methods
when learning the mapping from (a 5, a 4)T to w; this results
because the time-dependent variable x is a part of the input,
which loses the advantage of the correct dot-product assumption. In contrast, the two-step method GPR-2 with dot-product kernels perfectly reproduces the polynomial. On the other
hand, however, SVR-1 performs better than SVR-2.
For the MDN, we select the most probable VMP parameter as the output. Except for the GPR with dot-product kernels, the MDN outperforms all other methods.
Random-Obstacles Avoidance
To show whether the methods can scale to a more complex task than the one shown in Figure 1, we randomly
placed three obstacles in 2D space and asked for the collision-free trajectories with random starts and goals. To
collect demonstrations, a person drew curves connecting
random starts and goals without collisions with three randomly generated 2D balls on a tablet. Without any
instructions, the human demonstrations show multiple
modes and models.
The success of the task execution requires that the generated trajectory connects the start and goal without any collision

G
PR

-2

N
D
M

G
PR

-2

)

-2
R

(d
-1

G
PR

-1
G
PR

)
EK

R
SV

5
4
3
2
1
0

-1

Error

Learning Mapping for (a5, a4)T

(a)

N
M
D

(d
p)

G
PR

-2
PR

-2

)

-2
R
SV

-1

G
PR

-1
PR
G

)
EK

R
SV

(d
p)

5
4
3
2
1
0

-1

Error

Learning Mapping for (a5, a4, ..., a0)T

(b)
Figure 4. The models learned to fit a fifth-order polynomial,
showing the results for learning a mapping from (a) a part of the
coefficients to the MP parameters and (b) all coefficients to the
MP parameters. dp: dot-product kernel.

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

with randomly placed obstacles. Due to the task complexity
and existence of multiple modes and models, previous
approaches cannot achieve acceptable results. With a data set
with 100 demonstrations, the TP-GMM has a success rate of
only approximately 45% with five local frames (three for the
obstacles and two for the start and goal). Both one-step and
two-step methods with SVR and GPR perform worse, with a
success rate of less than 30%.
For the MDN, we assume three mixture components:
K = 3. To extract the latent feature (see the blue box in Figure 3), we introduce three separate network branches for
three obstacles. Each branch takes the position of one obstacle
and the start and goal of the trajectory as the input and outputs a hidden feature vector. The three hidden feature vectors
are then concatenated for the rest of the MDN. The entire
MDN is trained in an end-to-end manner.
During testing, we select the most probable mode, as mentioned previously. For each number of training data, 30 experiments are conducted for 30 different training data sets
randomly chosen from the collected demonstrations. To utilize the failure cost function, after each 100 training steps, the
MDN is evaluated on an evaluation data set to produce failed
samples. To avoid increasing data, we consider only the most
recent 3,000 failed samples.
As shown in Figure 5, with 100 demonstrations, the MDN
with both the entropy and failure cost functions
(l NLL + l model + l neg) achieves a success rate of approximately
82%. The performance is improved further to 85% with 300
demonstrations. With 100 demonstrations, the entropy MDN
with the failure cost function (l NLL + l model + l neg) achieves the
best result, and the entropy MDN (l NLL + l model) is better than
the original MDN (l NLL). Their difference decreases with an
increasing number of demonstrations.
In Figure 6 [16], testing samples are shown. The colorized
solid curves are generated by the most probable mode (MP
parameter) given by the MDN, and the transparent dashed
curves correspond to the other two modes, which are not
selected by the MDN, with relatively small output mixing coefficients. Three different colors indicate three different modes:
the green curves bend toward the top, red curves bend toward
the bottom, and blue curves connect the start and goal directly.
As can be seen, the MDN accomplishes the task with two steps.
One step is to separately update each mixture component
branch to increase the success rate of their modes. The other
step is to adjust the mixing-coefficient output to select the
mode that has the highest chance of accomplishing the task.
The Hit-the-Ball Experiment in MuJoCo
In this experiment, the robot hits the ball with its fist. After
being hit, the ball slides on the table and stops at some locations. The final location of the ball on the table is the task
parameter query. The purpose is to generate an appropriate
robot motion to hit the ball and let the ball stop at a specific
location. We conducted this experiment in the MuJoCo
simulator [20] with the model of the humanoid robot
ARMAR-6 [21].

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020