IEEE Robotics & Automation Magazine - June 2020 - 51

Two real and three simulated environments with different complexity levels are used, all employing raw images as
observations. The simulated environments are mountain-
car, swing-up pendulum, and car racing, the implementations of which are taken from OpenAI Gym [17]. These
simulations provide rendered image frames as observations of the environment. The frames visually describe the
position of the system but not its velocity, which is necessary to control the system. The experiments on the real
physical systems consist of a swing-up pendulum and a
setup for picking oranges on a conveyor belt with a three
degrees of freedom (3 DoF) robot arm. The metrics used
for the comparisons are the achieved final policy performance and the speed of convergence, which is very relevant when dealing with real systems and human teachers.
A video showing most of these experiments can be found
at https://youtu.be/4kWGfNdm21A.
Ablation Study
In this ablation study, the architecture of the network is the
independent variable. Three independent comparisons were
carried out using DAgger, HG-DAgger, and D-COACH.
The training sessions were run using a simulated teacher to
avoid any influence from human factors. Three different
architectures were tested for learning the policy from
an oracle. The structure of the networks is introduced in
the following:
1) Full network: the proposed architecture
2) Memoryless SRL (M-Less SRL): similar to the full network
but without using recurrence between the encoder and
decoder (the autoencoder is trained using the reconstruction error of the observation)
3) Direct policy learning (DPL): the same architecture as in the
full network but without using SRL, i.e., not training the
transition model (the encoding, recurrent layers, and policy are trained using only the cost of the policy).
The ablation study is done on a modified version of the car
racing environment. Normally, this environment provides an
upper view of a car on a race track. In this case, we occluded
the bottom half of the observation such that the agent was not
able to precisely know its position on the track. This position
can be estimated if past observations are taken into account.
As a consequence, this is an appropriate setting for making a
comparison of different NN architectures. Table 1 gives the
various performances obtained by the learning algorithms
when modifying the structure of the network. The results
show a normalized averaged return through 10 repetitions for
each experiment, in which five evaluations were carried out
for each of the repetitions.
As expected, DAgger with the full architecture obtained
the best performance, and, given that it received new samples
every time step, it was robust against changes in the architecture, even when it did not have memory. On the other hand,
D-COACH was very sensitive to changes in the architecture,
especially the DPL architecture. This shows how the full
model is able to enhance the performance of the agents in

Table 1. A comparison of the performance
(return) of different learning methods in the
car racing problem.
Full

M-Less SRL

DPL

D-COACH

0.97

0.76

0.68

DAgger

0.87

0.96

HG-DAgger

0.89

0.69

0.9

The returns were normalized with respect to the best performance
(DAgger full).

problems where temporal information is required. It even
makes D-COACH perform almost as well as DAgger, despite
the fact that the former does not require constant and perfect
teacher feedback. Finally, HG-DAgger was more robust than
D-COACH in the DPL case, but its performance with the full
model was not as good.
Simulated Tasks With Simulated Teachers
In the second set of experiments, we performed a comparison among the DAgger, HG-DAgger, and D-COACH
algorithms using the proposed full network architecture.
To keep the experiments free of human-factor effects, the
teaching process was, once again, performed with simulated teachers. The methods were tested in the mountain-
car (in the supplementary material) and swing-up
pendulum simulated problems. A mean of the return
obtained through 20 repetitions is presented for these
experiments, along with the maximum and minimum values of these distributions.
Swing-Up Pendulum
In the case of the swing-up pendulum, the results are very
different for both DAgger agents (see Figure 4), which
have a higher rate of improvement than D-COACH during
the first minutes, when the policy is learning the swinging
behavior. Since the swinging part requires large actions,
the improvement with D-COACH is slower. However,
once the policy is able to swing the pendulum up, the second part of the task is to keep the balance in the upright
position, which requires fine actions. It is at this point that
learning becomes easier for the D-COACH agent, which
obtains a constant and faster improvement than the HGDAgger agent, even reaching a higher performance. In
Figure 4, the expected performance upper bound is indicated by a black dashed line, which is the return obtained
by the simulated teacher. The purple dashed line shows
the performance of a random policy, which is the expected
lower bound.
Simulated Tasks With Human Teachers
The previous experiments give insights into how the policy
architectures and/or the learning methods perform when imitating an oracle. Most IL methods are intended for learning
JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

https://www.youtube.com/watch?v=4kWGfNdm21A&feature=youtu.be

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020