IEEE Robotics & Automation Magazine - June 2020 - 122

function in the previous simulator, whereas GPQ-MFRL
checks whether the agent has a sufficiently accurate estimate
of optimal Q values in the previous simulator (line 8). Lines
10-15 describe the main body of the algorithm, where the
agent records the observed transitions in D i . We update target values (line 14) for every transition, as more data are collected in D i (line 13). The GP model is updated after every
step (line 16).
The agent utilizes the experiences collected in higher simulators (lines 25-27) to choose the optimal action in the current simulator (line 6). Specifically, it checks for the
maximum-fidelity simulator in which the posterior variance
for (s, a) is less than a threshold v th . If one exists, it utilizes
the Q values from the highest known simulator to choose
the next action in the current simulator. If no such higher
simulator exists, the Q values from the previous simulator
(line 24) are considered to choose the next action in the current simulator with an additive fidelity parameter b .
GPQ-MFRL performs a batch retraining every time the
robot collects new sample in a simulator (lines 13-15). During batch retraining, the algorithm updates the target values
in previously collected training data using the knowledge
Goal State

gained by collecting new samples. Then, these updated target
values are used to predict the Q values using GPs (line 16). As
the amount of data grows, updating the GP can become computationally expensive; however, we can prune the data set
using sparse GP techniques [6]. It is nontrivial to choose values for confidence bounds, but, for the current experiments,
we chose the v sum
th to be 10% of the maximum Q value possible and v th to be one fifth of v sum
th .
Results
We use two environments to simulate GP-VI-MFRL and
three environments for GPQ-MFRL. For GP-VI-MFRL, the
goal was learning to navigate from one point to another
while avoiding the obstacles. R 1 is a 21 # 21 grid world
with a point robot, while R 2 is Gazebo (discretized in a 21
# 21 grid), which simulates the kinematics and dynamics
of a quadrotor operating in 3D. For GPQ-MFRL, the goal
was to learn to avoid the obstacles while navigating through
the environment. R 1 is the Python-based simulator Pygame, R 2 is a Gazebo environment, and R 3 is the real world.
We further used sparse GPs to speed up the computations
required to perform GP inference. We report the improvements in computational time and a direct comparison
between GP-VI-MFRL and GPQ-MFRL on an obstacleavoidance task.
GP-VI-MFRL Algorithm
The task of the robot is to navigate from the start state to goal state.
The start and goal states and the obstacles for the environment
used are shown in Figure 3. The state of the robot is given by its x
and y coordinates, and the action is a 2D velocity vector. Both
simulators have the same state space; therefore, t i is an identity
mapping. The robot gets a reward of zero for all transitions
except when it hits the obstacles (in which case it gets a reward
of −50), and it gets a reward of 100 for landing in the goal state.
Since the state space S ! R 2 and the action space
(velocity) A ! R 2, the true transition function is R 4 " R 2.

Start State

(a)
400
Number of Samples

350
300
250
200
150
100
50
0

(b)
Figure 3. The environment setup for a multifidelity simulator
chain. (a) The grid-world simulator (R 1) has two walls, while
(b) the Gazebo simulator (R 2) has four walls.

122

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

Grid World

Gazebo

Figure 4. The samples collected at each level of the simulator for
a 21 # 21 grid in the grid-world and Gazebo environments. The
values of v sum
and v th were kept at 0.4 and 0.1, respectively.
th

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020