IEEE Robotics & Automation Magazine - June 2020 - 47

feedback. To achieve this, we focus on robots interactively
learning these policies from non-expert humans who act as
teachers. We present a neural network (NN) architecture along
with an interactive imitation learning (IIL) method that efficiently learns spatiotemporal features and policies from raw
high-dimensional observations (raw pixels from an image) for
tasks in environments that are not fully temporally observable.
We denominate IIL as a branch of IL, where human teachers provide different kinds of feedback to robots, such as new
demonstrations triggered by robot queries [1], corrections [2],
preferences [3], reinforcements [4], and so forth. Most IL
methods work under the assumption of learning from perfect
demonstrations; therefore, they fail when teachers have only
partial insights about the task execution. Non-expert teachers
could include all users who are neither machine learning/control experts nor skilled enough to fully show the desired
behavior of the policy.
Interactive approaches, such as COACH (which is the
short form of Corrective Advice Communicated by Humans)
[5], and some interactive reinforcement learning (IRL)
approaches [4], [6] are intended for non-expert teachers but
are not completely deployable for end users. Sequential decision-making learning methods (IL, IIL, IRL, and so forth) rely
on good SRs, which simplify the shaping of the policy landscape and provide suitable generalization properties. However, this requirement means that experts on feature
engineering must preprocess the states properly before running the learning algorithms.
The inclusion of deep learning (DL) in IL (given the popularity DL has gained in the field of RL [7]) enables practitioners
to skip preprocessing modules for inputting policies since some
NN architectures endow the agents with intrinsic featureextraction capabilities. This has been exhaustively tested in endto-end settings [7]. In this regard, DL enables non-expert
humans to shape policies based only on their feedback.
Nevertheless, in real-world problems, we commonly face
tasks wherein the observations do not explain the complete
state of the agent, due to the lack of temporal information (e.g.,
rates of change) or because the agent-environment interaction
is non-Markovian (e.g., dealing with occlusions). For these
cases, it is necessary to provide memory to the learning policy.
Recurrent NNs (RNNs) can learn to model dependencies
from past observations and map them to the current outputs.
This recurrency has been used in RL and IL, mostly through
long short-term memory (LSTM) networks [8].
Therefore, LSTMs are included in our NN architecture to
learn temporal features, which contain relevant information from
the past. However, DL algorithms require large amounts of data;
as a way to tackle this shortcoming, SR learning (SRL) has been
used to learn features more efficiently [9], [10]. Considering that
real robots and human users have time limitations, as an SRL
strategy, a model of the world is learned to obtain SRs that make
the policy convergence possible within feasible training time
intervals (see Figure 1). The combination of SRL and the teacher's
feedback is a powerful strategy for efficient learning of temporal
features from raw observations in non-Markovian environments.

The experiments presented in this article show the impact
of the proposed architecture in terms of data efficiency and
the policy's final performance within the deep COACH
(D-COACH) IIL framework [11]. Additionally, the experimental procedure demonstrates that the proposed architecture could even be used with other IL methods, such as data
aggregation (DAgger) [12]. The code used in this paper can
be found at https://github.com/rperezdattari/Interactive
-Learning-of-Temporal-Features-for-Control.
Background and Related Work
Our method combines elements from SRL, IL, and memory
in NN models to build a framework that enables non-expert
teachers to interactively shape policies in tasks with nonMarkovian environments. These elements are introduced in
the following.
Dealing With Non-Markovian Environments
There are different reasons why a process could be partially
observable. One is when the state describes time-dependent
phenomena, but the observation contains only partial information about them. For instance, velocities cannot be estimated from camera images unless observations from different
time steps are combined. Other examples of time-dependent
phenomena are temporary occlusions and corrupted communication systems between the sensors and the agent.
For these environments, the temporal information needs
to be implicitly obtained within the policy model. There are
two well-known approaches for adding memory to agents in
sequential decision-making problems when using NNs as
function approximators:
World Model
Agent

Feedback ht
Observation ot

Action at
Teacher

Environment
Figure 1. Interactively shaping policies with agents that model
the world. (Source: turkkub and Freepik from Flaticon.)

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

https://www.github.com/rperezdattari/Interactive-Learning-of-Temporal-Features-for-Control https://www.github.com/rperezdattari/Interactive-Learning-of-Temporal-Features-for-Control

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020