IEEE Robotics & Automation Magazine - June 2020 - 121

to learn the optimal policy. For a given state-action pair (s, a),
the estimated transition function is defined by a normal distribution, with the mean and variance given by (2) and (3). Algorithm 1 gives the details of the proposed framework.
Before executing an action, the agent checks (line 8)
whether it has a sufficiently accurate estimate of the transition
function for the current state-action pair in the previous simulator, R i - 1 . Specifically, we check whether the variance of
the current state-action pair in the previous simulator is less
than v th . If not, and if the transition model in the current
environment has changed, it switches to R i - 1 and executes
the action in the potentially less expensive environment. The
agent lands in the state ti (s) in the lower-fidelity simulator.
We also keep track of the variance of the L most recently visited state-action pairs in the current simulator. If the
running sum of the variances is below a threshold (line 15),
this suggests that the robot is confident about its actions in
the current simulator and can advance to the next one. In
the original work [2], the agent switched to the higher-fidelity simulator after a certain number of known state-action
pairs were encountered. In our implementation (line 7), the
model of the current environment changes if the posterior
variance for a state-action pair drops below a threshold
value (i.e., the agent has a sufficiently accurate estimate of
the transitions from that state). Lines 10-13 describe the
main body of the algorithm, where the agent executes the
greedily chosen action and records the observed transition
in D i . The GP model for the transition function is updated
after every step (line 12). New Q value estimates are computed every time after an update of the transition function
(line 13). Note that we use a separate GP to estimate the
transition function in each simulator.
One can use a number of termination conditions (line 5),
e.g., maximum number of steps, changes in the value function, or maximum number of switches. In our implementation, Algorithm 1 terminates if the change in new estimates
of value functions in the real-world environment is no
more than a certain threshold (10%) compared with the
previous estimates.
The planner utilizes knowledge of transitions from higher
(lines 26-28) and lower simulators (line 25) to encourage
exploration in the current simulator. For every state-action
pair (s, a), the planner looks for the maximum-fidelity simulator in which a known estimate of transitions for (s, a) is
available and uses them to plan in the current simulator. An
estimate is termed known if the variance is below a threshold.
If no such simulator is available, then it uses the Q values
learned in the previous simulator plus a fidelity parameter b .
This parameter models the maximum possible difference
between the optimal Q values in consecutive simulators. The
higher-fidelity simulator values will always be trusted over the
lower-fidelity ones as long as we have low enough uncertainty
in those estimates. We assume that, for two consecutive simulators, the maximum difference between the optimal action
value of a state-action pair in Ri and the corresponding pair
in R i - 1 is not more than b i . One must apply a state-space

discretization to plan the actions. However, the learned transition function is continuous.
GPQ-MFRL Algorithm
The agent learns optimal Q values using GPs directly instead
of learning the model first. The underlying assumption is that
nearby state-action pairs produce similar Q values. This
assumption can also be applied to problems where the states
and actions are discrete but the transition function implies
some sense of continuity. We chose the squared-exponential
kernel because it models the spatial correlation we expect to
see in a ground robot. However, any appropriate kernel can be
chosen. We used a separate GP per simulator to estimate the
Q values from data collected only in that simulator.
Algorithm 2 gives the details of the proposed framework.
GPQ-MFRL continues to collect samples in the same simulator until the agent is confident about its optimal actions. If the
running sum of the variances is below a threshold (line 17),
this suggests that the robot has found a good policy with high
confidence in the current simulator, and it must advance to the
next one (line 18).
GPQ-MFRL uses similar thresholds (v th and v sum
th ) as
GP-VI-MFRL to determine when to switch to a lower- or
higher-fidelity simulator. GP-VI-MFRL checks whether the
agent has a sufficiently accurate estimate of the transition

Algorithm 2: GPQ-MFRL
1: procedure
2: Input: confidence parameters v th and v sum
th ; simulator
chain GR, fidelity parameters b, state mappings tH; L.
t i = initialize GP for i ! " 1, f, d ,; state s 0 in
3: Initialize: Q
simulator R 1; i ! 1; change = False.
4: Initialize: t ! 0; D i ! {} for i ! " 1, f, d , .
5: while terminal condition is not met
6:
a t ! CHOOSEACTION(s t, i )
7:
if v i (s t, a t) # v th : change = True
8:
if v( t i (s t), a t) 2 v th and change and i 2 1
s t ! t i (s t), i ! i - 1, continue
9:
Grt, s t + 1H ! execute action a t in R i
10:
appendG s t, a t, s t + 1, rtH to D i
11:
Yi ! {}
12:
13:
for G s t, a t, s t + 1, rtH ! D i // batch training //
t i (s t + 1, a)
y t ! rt + c max a Q
14:
append G s t, a t, y tH to Yi
15:
t i ! update GP i using Yi
Q
16:
j = t-L
17:
if / j = t v i (s j, a j) # v sum
and t 2 L and i 1 d
th
18:
s t ! t -i +11(s t), i ! i + 1
19:
change = False
20: end procedure
21:
22: procedure CHOOSEACTION(s, i )
23: for a ! A(s)
t i - 1( t i (s), a) + b i
Q(s, a) = Q
24:
25:
for k ! " i, f, d ,
26:
s k = t -k 1 f t -i +12 t -i +11(s)
t k(s k, a)
27:
if v k(s k, a) # v th : Q(s, a) = Q
28: return arg max a Q(s, a)
29: end procedure

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

121
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
