IEEE Robotics & Automation Magazine - June 2020 - 49

! i - a $ d i J (i).

(1)

The feedback given by the human indicates only the sign of
the policy error. Its magnitude is supposed to be unknown
since the algorithm works under the assumption that the user
is non-expert; therefore, she/he does not know the magnitude
of the proper action. Instead, the error magnitude is defined as
the hyperparameter e that must be defined before starting the
learning process. Thus, the policy errort is defined by h t $ e.
To compute a gradient in the parameter space of the policy, the error needs to be a function of i. This is achieved by
observing that
errort (i) = a target
- r i (o t),
t

(2)

where a target
is the incremental objective generated by the
t
feedback of the human a target
= a t + errort, and a t is the
t
r
.
current output of the policy i From (1), (2), and the derivative of the mean squared error, we can get the COACH
update step:
i

! i + a $ errort $ d i r i .

(3)

To be more data efficient and avoid locally overfitting to the
most recent corrections, D-COACH has a memory buffer that
stores the tuple (o t, a target
) and replays this information during
t
learning. Additionally, when working in problems with highdimensional observations, an autoencoding cost is incorporated in D-COACH as an observation reconstruction SRL
strategy. In the D-COACH pseudocode (Algorithm 2), this
SRL step is omitted. D-COACH learns everything from
scratch through only one interactive phase, unlike other deep
interactive RL approaches [4], [6], which split the learning
process into two sequential learning phases: first, recording
samples of the environment for training a dimensionality
reduction model (e.g., an autoencoder) and, second, using that
model for the input of the policy network during the actual
interactive learning process.

finding compact Markovian embeddings. We propose a neural
architecture separated into two parts: 1) the transition model
and 2) the policy. The transition model is in charge of learning
the dynamics of the environment in a supervised manner,
using samples collected by the agent. The policy part is shaped
using only corrective feedback. Figure 2 shows this architecture.
Learning to predict the next observation o t +1 forces a
Markovian SR. This has been successfully applied in RL [16].
RNNs can encode information from past observations in
. Thus, the objective of the first part of
their hidden state h LSTM
t
u t +1, which, as a conM
(
o t, a t, h LSTM
the NN is to learn
t-1 ) = o
. Addisequence, learns to embed past observations in h LSTM
t
tionally, when the observations are high-dimensional (raw
images), the agents also need to learn to compress spatial
information. To achieve this, a common approach is to compress this information in the latent space of an autoencoder.
For the first part of the architecture, we propose a combination of an autoencoder with an LSTM to compute the

Algorithm 2: D-COACH
1: Require: error magnitude e, buffer update interval b
2: Init: B = [] # initialize memory buffer
3: for t = 1, 2, f do
4:
observe state o t
5:
execute action a t = r i (o t)
6:
feedback human corrective advice h t
7:
if h t is not 0 then
error t = h t $ e
8:
9:
a target (t) = a t + error t
target
10:
update r using SGD with pair (o t, a t )
11:
update r using SGD with a minibatch sampled
from B
target
12:
append (o t, a t ) to B
13:
if mod(t, b) is 0 then
14:
update r i using SGD with a minibatch sampled
from B

Learning Temporal Features Based on
Interactive Teaching and World Modeling
In this section, the SRL NN architecture is described along
with the interactive algorithm for policy shaping.

LSTM

~
ot+1

Transition
Model

Network Architecture for Extracting
Temporal Features
When approaching problems that lack temporal information
in the observations, the most common solution is to model
the policy with RNNs, as discussed in the "Dealing With
Non-Markovian Environments" section; therefore, we propose to shape policies that are built on top of RNNs, with
occasional human feedback. In this article, we use the terms
world model and transition model interchangeably.
IIL methods can take advantage of SRL for training with
other objective functions by 1) making use of all of the experience collected in every time step and 2) boosting the process of

LSTM

ht-1

Policy

Figure 2. The general structure of the transition model and
policy.

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020