IEEE Robotics & Automation Magazine - June 2020 - 50

Transition Model
e (ot)

Encoder

CC2

CC1
C1

FC1

R1
FC2

LSTM
ht-1

htLSTM

Memory
FC3

~
ot +1

Decoder
N3

FC4

DC1

DC2

DC3

Convolution:
Deconvolution:
Reshape:
Normalization:
Fully Connected:
Recurrent:

FC5

FC6

FC7

Policy

Identify:
Concatenate:

Figure 3. The proposed NN architecture. Convolutional and recurrent (LSTM) layers are included in the transition model for learning
of spatiotemporal SRs. The estimated state st t is used as input to the policy, which is a fully-connected NN. C: convolution;
N: normalization; FC: fully connected; CC: concatenate; R: recurrent; DC: deconvolution. (Source: Vectors Market, Smashicons,
Freepik, and Smalllikeart from Flaticon.)

Algorithm 3: Online temporal feature learning
1: Require: Policy update algorithm r update, training
sequence length x, model update rate d
2: Init: D = []
3: for t = 1, 2, f do
4:
observe observation o t
5:
append (o t -1, f, o t - x, a t -1, f, a t - x, o t) to D
6:
execute action a t = r i(o t, h LSTM
t -1 )
7:
compute h LSTM
from M (o t, a t, h LSTM
t
t -1 )
8:
feedback human feedback h t
9:
call r update(o t, a t, h t)
10:
if mod(t, d ) is 0 then
11:
update M using SGD with minibatches of
sequences sampled from D

transition function, i.e., predicting the next high-dimensional observation. A detailed diagram of this architecture can
be seen in Figure 3. In the second part of the architecture,
the policy takes as input a representation of the state ts t,
which is generated inside the transition model network.
This representation is obtained at the output of a fully connected layer (FC3) that combines the information of h LSTM
t -1
with the encoder compression of the current observation
e (o t). This is achieved by adding a skipping connection
between the output of the encoder and that of the LSTM.
Interactive Algorithm for Policy and
World-Model Learning
Algorithm 3 presents the pseudocode of the SRL strategy.
The hidden state of the LSTM is denoted as h LSTM and the
human corrective feedback as h. In every time step, a buffer
D stores the samples of the transitions with sequences of
length x (line 5). The agent executes an action based on its
50

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

last observation and the current hidden state of the LSTM
(line 6). This hidden state is updated using its previous
value and the most recent observation and action (line 7).
Line 8 captures the occasional feedback of the teacher,
which could be a relative correction when using D-COACH
or a corrective demonstration when using HG-DAgger.
Also, depending on the learning algorithm, the policy is
updated in different ways (line 9). D replays past transitions of the environment to update the transition function
model (line 11). This is done following the bootstrapped
random updates [13] strategy. This model is updated every
d time steps.
Experiments and Results
In this section, we present experiments for validating the
proposed NN architecture and the interactive training
algorithm. To obtain a thorough evaluation, different
experiments are carried out to compare and measure the
performance (the return, i.e., the sum of the rewards) of
the proposed components. Initially, the network architecture based on SRL is evaluated in an ablation study
aiming to quantify the data efficiency improvement
added by the network architecture's different components. Then, using the proposed architecture, D-COACH
is compared with HG-DAgger using simulated tasks
and simulated teachers (oracles). The third set of experiments is performed with human teachers in simulated
environments, again comparing different learning methods. Finally, a fourth set of validation experiments is
conducted in real systems with human teachers. Most of
the results are presented in this article; however, some
are in the supplementary material, which can be found
in IEEE Xplore along with more detailed information
about the experiments.

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020