IEEE Robotics & Automation Magazine - March 2016 - 97

System identification methods can be used to build a parameterized model, and the acquired model can virtually sample
the data for policy improvement [1], [4], [14], [17], [22], [35],
[36]. However, for policy improvement, the learned simulation
model's generalization performance needs to be carefully verified. Even though the simulation model can be useful for predicting state transitions, explicitly predicting them is not
necessary for policy updates.
In this article, we propose using previously acquired data
as a simulation model instead of building a parameterized
simulation model. To utilize the previously acquired data to
improve the current policy, we need to reevaluate the previous
data in terms of the current policy. To do this in reinforcement-learning (RL) frameworks, importance-weighted policy
gradients with parameter-based exploration (IW-PGPE) can
be used [34]. The usefulness of this approach has been thoroughly evaluated by comparisons in numerical simulations
with previously proposed RL methods [6], [9], [15], [25], [29].
However, no work has clearly stated and shown that this
particular combination of PGPE and importance weighting
to derive the gradient of objective functions is suitable for
real robot learning. In our previous study, we presented
preliminary results and showed that IW-PGPE is a useful
approach for humanoid motor learning [23]. In this article,
we extend our IW-PGPE algorithm to more efficiently
reuse previous experiences by introducing a recursive operation to policy updates and show how our extended algorithm is useful for real robot learning in high-dimensional
spaces. We successfully applied our proposed approach to
two different tasks with two different conditions. First, we
applied it to a cart-pole swing-up task in a real-virtual hybrid environment with a PS Move motion controller. Then
we applied it to a challenging basketball-shooting task in a
real environment.

number of steps, called the horizon length. We assume that T is
a fixed deterministic number. Then the discounted cumulative
reward along h, called the return, is given by
R (h) :=

T-1

/ c t - 1 r (x (t), u (t)) + U (x (T)),

(1)

t=1

where c ! [0, 1) is the discount factor for future rewards. The
immediate and terminal rewards are r ^x ^ t h, u ^ t hh and
U (x (T)).
Policy Models
In this article, we consider feedback and feedforward policy
models.
Feedback Policy Model
We use locally linear state-dependent basis functions in our
feedback policy model [20], [21], [31]:
u (t) = W fb z fb (z (t)),

(2)

where z (t) ! Z is a feedback state at time t. Note that state
space Z can be a subset of the original state space, i.e., Z 1 X.
W fb is a state-dependent matrix, and z fb (z (t)) ! 0 M is a vector that consists of state-dependent basis functions, where M is
the number of basis functions.
Feedforward Policy Model
The feedforward policy model is formulated as:
u (t) = W ff z ff (t),
M

(3)
ff

M#1

where w ! 0 is the parameter vector, z (t) ! 0
is a
vector that consists of time-dependent basis functions, and
W ff is a parameter matrix.

Motor Learning Framework
Trajectories and Returns
We assume that the underlying control problem is a discretetime Markov decision processes. At each discrete time step t,
the agent observes state x (t) ! X, selects action u (t) ! U,
and receives immediate reward r (t) of the results from a state
transition in the environment. The environment's dynamics
are characterized by p (x (t + 1) | x (t), u (t)), which represents the transition probability density from current state x (t)
to subsequent state x ^t + 1h when action u (t) is taken;
p ^x ^ 1 hh is the probability density of the initial states. Immediate reward r (t) is given based on reward function
r ^x ^ t h, u ^ t h, x ^t + 1hh .
The robot's decision-making procedure at each time step t is
characterized by parameterized policy p ^u ^ t h x ^ t h, w) with
parameter w, which represents the conditional probability density of taking action u (t) in state x (t). We assume that the policy is continuously differentiable with respect to parameter w.
A sequence of states and actions forms a trajectory denoted
by h := [x (1), u (1), f, x (T), u (T)], where T denotes the

Table 1. The supplemental equations for low-level
controller and gradient of objective function with
reference to policy parameters.
1.1) The expected return in the PGPE formulation is defined
in terms of the expectations over both h and w as a function of hyperparameter t:
J (t): =

##

p (h ; w) p (w ; t) R (h) dhdw.

1.2) The derivative of the expected return using log arithmic
d t p (w ; t)
:
derivative d t log p (w ; t) =
p (w ; t)
d t J (t) =

##

p (h ; w) p (w ; t) d t log p (w ; t) R (h)dhdw.

1.3) The expectations over h and w are approximated by
empirical averages:
N
d t Jc (t) = 1 / d t log p (w n ; t) R (h n).
N n=1
2) We can acquire the gradient information for policy updates
that are weighted by importance weight v:
Nl

d t tJ IW (t) := 1 / v (w ln) d t log p (w ln ; t) R (hln).
Nl n = 1

march 2016

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

97
http://www.d.tt
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2016
