IEEE Robotics & Automation Magazine - June 2020 - 48

1) Observation stacking policies [7]: stacking the last N observations (o t, o t - 1, f o N ) and using this stack as the input of
the network
2) Recurrent policies [13]: including RNN layers in the policy
architecture.
One of the main issues with observation stacking is that
the memory of these models is determined by the number of
stacked observations. The overhead increases rapidly for larger sequences in high-dimensional observation problems.
In contrast, RNNs can model information for an arbitrarily
long period of time [14]. Also, they do not add input-related
overheads because, when these models are evaluated, they use
only the last observation. Therefore, RNNs have a lower computational cost than observation stacking. Given the more
practical usage of recurrent models and their capability of representing arbitrarily long sequences, in this article, we use
RNN-based policies (with LSTM layers) in the proposed NN
architecture. Nevertheless, the use of LSTMs has a critical disadvantage since their training is more complex and requires
more data, something very problematic when considering
human teachers and real systems. We now introduce SRL,
which helps to accelerate LSTM convergence.
SRL
In most of the problems faced in robotics, the state s t, which
fully describes the situation of the environment at time step t,
is not fully accessible from the robot's observation o t . As
mentioned, in several problems these observations lack the
temporal information required in the state description. Moreover, these observations tend to be raw sensor measurements
that can be high-dimensional, highly redundant, and ambiguous. A portion of this data may even be irrelevant.
As a consequence, to successfully solve these problems a
policy needs to 1) find temporal correlations between several
consecutive observations and 2) extract relevant features
from observations that are hard to interpret. However, finding relations among these large data structures with the
underlying phenomena of the environment while also learning controllers can be extremely inefficient. Therefore, efficiently building controllers on top of raw observations
requires learning-informative, low-dimensional SRs [15].
The objective of SRL is to obtain an observer capable of generating such representations.
Algorithm 1: (HG-)DAgger
1: Require: demonstrations database D with initial
demonstrations, policy update frequency b
2: for t = 1, 2, f do
3:
if mod(t, b) is 0 then
4:
update r i from D
5:
observe state o t
6:
select action from agent or expert
7:
execute action
8:
feedback provide label a )t for o t, if necessary
9:
aggregate (o t, a )t ) to D

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

A compact representation of a state is considered suitable
for control if the resulting SR
● is Markovian
● has good generalization to unseen states
● is defined in low-dimensional space (considerably lower
than the actual observation dimensionality) [9].
Along with the control objective function (e.g., reward
function and imitation cost function), other objective functions can be used for SRL [10]:
● observation reconstruction
● forward model or next observation prediction
● the inverse model
● the reward function
● the value function.
Interactive Learning Methods
This section briefly introduces two approaches for interactively learning from human teachers while agents are executing a task.
Data Aggregation: DAgger and Human-Gated DAgger
DAgger [12] is an IIL algorithm that aims to collect data
through online sampling. To achieve this, trajectories are generated by combining the agent's policy r i and the expert's
policy. The observations o t and the demonstrator's corresponding actions a )t are paired and added to a database D,
which is used for training the policy's parameters i iteratively
in a supervised learning manner to asymptotically approach
the expert's policy. At the beginning of the learning process,
the demonstrator has all the influence over the trajectory
made by the agent; then the probability of following the demonstrator's actions decays exponentially.
For working in real-world systems with humans as demonstrators, a variation of DAgger, human-gated DAgger
(HG-DAgger) [2], was introduced. In this approach, the
demonstrator is not expected to give labels across every action
of the agent but only in places where she/he considers that the
agent's policy needs improvement. Only these labels are
aggregated to the database and used for updating the policy.
Additionally, every time feedback is given by the human, the
policy will follow the provided action. As a safety measure,
in HG-Dagger, the uncertainty of the policy across the
observation space is estimated; that element is omitted in
this article. Algorithm 1 shows the general structure of
DAgger and HG-DAgger.
D-COACH
In this framework, humans shape policies, giving occasional
corrective feedback for actions executed by the agents [11].
The human indicates agent actions that she/he considers to be
erroneous through a binary signal h t, the direction in which
the action should be modified. This feedback is used to generate an error signal for updating the policy parameters i. It is
performed in a supervised learning manner, with the cost
function J and using the mean squared error and stochastic
gradient descent. Hence, the update rule is

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020