IEEE Robotics & Automation Magazine - June 2020 - 119

Multiple approximations are available that estimate the same
output with varying accuracy and costs. More generally, K
different fidelity approximations, f (1), f, f (K ), are available
representing the relationship between the input and output,
f (i) : X " Y, i = 1, f, K. Obtaining observations from the
ith approximation incurs cost c (i), and typically c (i) 1 c ( j)
for i 1 j.
There has been a significant surge in multifidelity methods
research following seminal work on autoregressive schemes.
Kennedy and O'Hagan [9] used GPs to explore ways in which
runs from several levels of a computer code can be used to
make inferences about the output of the complex computer
code. A complex code approximates reality better, but, in
extreme cases, a single run of a complex code may take many
days. The GP framework is a natural candidate for estimating
a phenomenon by combining data from various fidelity
approximations [10], [11].
Multifidelity methods have been widely used in RL applications to automatically optimize the parameters of control
policies based on data from simulations and experiments.
Cutler et al. [2] introduced a framework that specifies the
rules concerning when to collect observations from different
fidelity simulators. Marco et al. [12] introduced a Bayesian
optimization algorithm that uses entropy [13] as a metric to
decide which simulator to collect the observations from. The
authors employed their algorithm for a cart-pole setup with a
Simulink model as the simulator.
Two closely related techniques addressing sim2real are
domain randomization and domain adaptation. In both
cases, the simulators can be controlled via a set of parameters. The underlying hypothesis is that some unknown set of
parameters closely matches the real-world conditions. In
domain randomization, the simulator parameters are randomly sampled, and the agent is trained across all of the
parameter values. In domain adaptation, the parameter values are updated during learning.
However, the goal in these methods is to reduce the reality gap using a parameterized simulator [1], [14]. This
restricts the use of such approaches to scenarios where altering the parameters for a simulator itself is trivial. Furthermore, only one type of simulator is used. In contrast, MFRL
techniques leverage multiple simulators with varying fidelity
levels and costs to operate. In addition, in MFRL, the policy
learned in the highest-fidelity simulator (real-world) uses
data from the same environment, unlike sim2real. As such,
these approaches are beneficial when there is a significant
reality gap, where maneuvers learned in simulators may not
translate to the real world.
Our work is inspired by the work of Cutler et al. [2], which
allows for bidirectional transfer of information between simulators. However, they used a tabular representation of the values function. We introduce two algorithms, in a similar spirit,
that can decide from which simulator to collect the observations. We hypothesize that using a GP with an MFRL framework will lead to further improvements in the number of
samples required from the real world.

Background
RL
RL problems can be formulated as a Markov decision process:
M = GS, A, P, R, cH, with state space S, action space A,
transition function P(s t, a t, s t + 1) 7 [0, 1], reward function
R (s t, a t) 7 R , and discount factor c ! [0, 1). A policy
r : S " A maps states to actions. Together with the initial state
s 0, a policy forms a trajectory g = " [s 0, a 0, r0], [s 1, a 1, r1], f,
where a t = r(s t). rt and s t + 1 are sampled from the reward
and transition functions, respectively.
We consider a scenario where the goal is to maximize
the infinite-horizon, discounted reward starting from a state
s 0 . The value function for state s 0 is defined as
V r(s 0) = E[/ tt == 03 c t rt (s t, a t)| a t = r (s t)]. The state-action
value function or Q value of each state-action pair
t
under policy r is defined as Q r(s, a) = E[/ tt == 3
0 c rt +1
(s t + 1, a t + 1)| s 0 = s, a 0 = a]. which is the expected sum of discounted rewards obtained starting from state s, taking action
a and following r thereafter. The optimal Q value function
Q ) for a state-action pair (s, a) satisfies Q )(s, a) =
max r Q r (s, a) = V )(s) and can be written recursively as
Q )(s t, a t) = E s t + 1[r (s t, a t) + cV )(s t + 1)].

(1)

Our objective was to find the optimal policy r )(s) =
argmax a Q )(s, a) when R and P are not known to the agent.
In model-based approaches, the agent learns R and P first and
then finds an optimal policy by calculating optimal Q values
from (1); the most commonly used model-based approach is VI
[15], [16]. We can also directly estimate the optimal Q values,
often known as model-free approaches [17], or directly calculate
the optimal policy, often known as policy-gradient approaches
[18]. The most commonly used model-free algorithm is
Q-learning. For the GP-VI-MFRL implementation, we use GPs
to estimate the transition function and VI to calculate the optimal policy. For our GPQ-MFRL implementation, we use
Q-learning to perform the policy update using GP regression.
In this article, two versions (model based and model free)
of GP-based MFRL are introduced. It should be remembered
that, although model-based algorithms are generally more
sample efficient in comparison to model-free algorithms, they
are not memory efficient [16]. Hence, depending on the
application, the model-free approach (GPQ-MFRL) may be a
suitable alternative to the more sample-efficient model-based
GP-VI-MFRL.
GPs
GPs are Bayesian nonparametric function approximators.
They can be defined as a collection of infinitely many random
variables, any finite subset X = " x 1, f, x k , of which is jointly Gaussian, with mean vector m ! R k and covariance matrix
K ! R k # k [19].
Let X = " x 1, f, x k , denote the set of training inputs, and
let y = " y 1, f, y k , denote the corresponding training outputs. GPs can be used to predict the output value at a new test
JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

119
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
