IEEE Robotics & Automation Magazine - June 2020 - 118

real-world training samples, which can be expensive and,
potentially, dangerous. In particular, obtaining exploratory
samples-which are crucial to learning optimal policies-
may require the robot to collide or fail, which is undesirable. Motivated by these scenarios, we focus on minimizing
the number of real-world samples required for learning
optimal policies.
One way to reduce the number of real-world samples is
to leverage simulators [1]. Collecting learning samples in a
robot simulator is often inexpensive and fast. One can use a
simulator to learn an initial policy, which is then transferred
to the real world-a technique usually referred to as sim2real [1]. This lets the robot avoid learning from scratch in the
real world, hence reducing the number of physical interactions required. However, this comes with a tradeoff.
Although collecting learning samples in simulators is inexpensive, they often fail to capture real-world environments
perfectly, a phenomenon called the reality gap. Although
simulators with increasing fidelity with respect to the real
world are being developed, one would expect there to
always remain some reality gap.
In this article, we leverage the concept of the MFRL algorithm [2], which uses multiple simulators with varying fidelity levels to minimize the number of real-world (i.e.,
highest-fidelity simulator) samples. The simulators, denoted
by R 1, f, R d, have increasing levels of fidelity with respect to
the real environment. For example, R 1 can be a simple simulator that models only the robot kinematics, R 2 can model
the dynamics as well as kinematics, and the highest-fidelity
simulator can be the real world (Figure 1).
MFRL differs from transfer learning [3], where a transfer
of parameters is allowed only in one direction. The MFRL
algorithm starts in R 1. Once it learns a sufficiently good policy in R 1, it switches to a higher-fidelity simulator. If it
observes that the policy learned in the lower-fidelity simulator
is no longer optimal in the higher-fidelity simulator, it switches back to the lower-fidelity simulator. Cutler et al. [2] showed
that the resulting algorithm has polynomial sample complexity and minimizes the number of samples required for the
highest-fidelity simulator.
The original MFRL algorithm learns the transition and
reward functions at each level. The reward and transition for
each state-action pair are learned independently of others.
Although this is reasonable for general agents, when planning

Gridworld

Robot Simulator

Real-World
Robot

Figure 1. The MFRL framework: the first simulator captures only
grid-world movements of a point robot, whereas the second
simulator has more fidelity, modeling the physics as well. The
control can switch back and forth between simulators and the
real environment, which is the third simulator.

118

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020

for physically grounded robots, we can exploit the spatial correlation between neighboring state-action pairs to speed up
the learning.
Our main contribution is to leverage the GP regression as
a function approximator to speed up learning in the MFRL
framework. GPs can predict the learned function value for
any query point and not just for a discretized state-action
pair. Furthermore, GPs can exploit the correlation among
nearby state-action values by an appropriate choice of a kernel. GPs have been extensively used to obtain optimal policies
in simulation-aided RL [4]. We take this further by using GPs
in the MFRL setting.
Other function approximators have been used in RL previously. We chose to use GPs since they require fewer samples
to learn a function when a good prior exists [5]. The priors
can be imposed, in part, by using appropriate kernels, which
make GPs flexible. A major limitation of learning with GPs is
their computational complexity, which grows cubically with
respect to the number of training samples. However, this issue
can be mitigated by using sparse approximations for GPs [6].
In MFRL, the state space of Ri is a subset of the state space of
R j for all j 2 i. Therefore, when the MFRL algorithm switches from Ri to R i + 1 , it already has an estimate for the transition function and Q values for states in R i + 1 . Hence, GPs are
particularly suited for MFRL, which we verify through our
simulation results.
Our main contributions in this article include introducing
● a model-based MFRL algorithm, GP-VI-MFRL, which
estimates the transition function and subsequently calculates the optimal policy using value iteration (VI)
● a model-free MFRL algorithm, GPQ-MFRL, which directly estimates the optimal Q values and, subsequently, the
optimal policy.
We verify the performance of the algorithms presented
through simulations and experiments with a ground robot.
Our empirical evaluation shows that the GP-based MFRL
algorithms learn the optimal policy faster than the original
MFRL algorithm with even fewer real-world samples.
Related Work
Multifidelity methods are prominently used in various
engineering applications to construct a reliable model of a
phenomenon when obtaining direct observations of that
phenomenon are expensive. The assumption is that we
have access to cheaply obtained but possibly less accurate
observations from an approximation of that phenomenon.
Multifidelity methods can be used to combine those observations with expensive but accurate observations to construct a model of the underlying phenomenon [4]. For
example, learning the dynamics of a robot using real-world
observations may cause wear and tear of the hardware [7].
Instead, one can obtain observations from a simulator that
uses a perhaps crude approximation of the true robot
dynamics [8].
Let f : X " Y denote a function that maps the input
x ! X 1 R d to an output y ! Y 1 R dl, where d, d l ! N.
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
