IEEE Robotics & Automation Magazine - June 2020 - 52

from any source of expert demonstrations. The source does
not necessarily have to be a human; it can be any type of
agent. However, the focus of this article is on learning from
non-expert human teachers, who are complex to model and
simulate. Therefore, conclusions have to be based on results
that also include validation with real users.
Experiments with the mountain-car (in the supplementary material) and swing-up pendulum were run with eight
human teachers. In this case, the classical DAgger approach
was not employed since, as discussed in the "Interactive
Learning Methods" section, it is not specifically designed for
human users. Instead, HG-DAgger was validated.
Swing-Up Pendulum
This task is relatively simple from a control theory point
of view. Nevertheless, it is quite challenging for humans

−500

Return

−1,000
−1,500
−2,000
−2,500
−3,000
−3,500

1,000 2,000 3,000 4,000 5,000 6,000 7,000
Time Step

Oracle
DAgger

5 6 7
Time (min)

Random Policy
HG-DAgger

9 10 11
D-COACH

to teleoperate the pendulum due to its fast dynamics.
Indeed, participants were not able to successfully teleoperate the agent; therefore, unlike the mountain-car
task, we could consider the participants as non-experts
in the undertaking.
Figure 5 displays the results of this experiment, which are
similar to the ones presented in Figure 4. At the beginning,
D-COACH has a slower improvement when learning to
swing up; however, it learns faster than HG-DAgger when the
policy needs to learn the accurate task of balancing the pendulum. For users, it is more intuitive and easier to improve
the balancing with the relative corrections of D-COACH than
with the perfect corrective demonstrations of HG-DAgger, as
users do not need to know the right action but, rather, just the
direction of the correction. Unlike the performance of the
simulated teacher depicted in Figure 4, the plot in Figure 5
shows the performance of the best human teacher teleoperating the pendulum with the same interface used for the teaching process. It can be seen that using both agents facilitated
obtaining policies that outperformed the non-expert
human teachers. All policies trained with D-COACH were
able to balance the pendulum, whereas, with HG-DAgger,
the success rate was half as high. Additionally, after the
experiment, the participants were queried about which
learning strategy they preferred. Seven out of eight
expressed a preference for D-COACH.
Validation on Physical Systems
With Human Teachers
The previous experiments performed comparison studies of
the NN architectures and learning methods under controlled conditions in simulated environments. In this
section, D-COACH is validated with human teachers and

Figure 4. The D-COACH and HG-DAgger comparison in the
swing-up pendulum problem using a simulated teacher.

−500
−1,000
Return

−1,500
−2,000
−2,500

−3,000
−3,500

2,000

4,000
6,000
Time Step

8,000

10,000
3

0 1 2 3 4 5 6 7 8 9 1011 1213 1415 16
Time (min)
Human (Teleoperation)
D-COACH

Random Policy
HG-DAgger

Figure 5. The simulated swing-up pendulum learning curve with
human teachers.

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

Figure 6. The orange-selector experimental setup. The 1:
conveyor belt; 2: orange samples; 3: frame observed by the
camera; 4: RGB camera; and 5: 3-DoF robot arm.

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020