IEEE Robotics & Automation Magazine - June 2020 - 123

However, generally, GP regression allows for single-dimensional outputs only. Therefore, we assume independence
between the two output dimensions and separately learn
two components (along x and y) of the transition functions,
x i + 1 = fx (x i, y i, a x) and y i + 1 = f y (x i, y i, a y), where (x i, y i)
and (x i + 1, y i + 1) are the current and next states of the robot, and (ax, a y) is the velocity input. The GP prediction is
used to determine the transitions, (x i, y i, a x) " x i + 1 and
(x i, y i, a y) " y i + 1, where (x i + 1, y i + 1) is the predicted next
state with variances v 2x and v 2y, respectively.
Figure 4 shows the switching between the simulators
pictured in Figure 3 for one run of the GP-VI-MFRL algorithm. Unlike unidirectional transfer-learning algorithms,
the GP-VI-MFRL agent switches back and forth between
the simulators, initially collecting most samples in the first
simulator. Eventually, the robot starts to collect more samples in the higher-fidelity simulator. This is the case when
the algorithm is near convergence and has accurate estimates for transitions in the lower-fidelity simulator as well.

Next, we describe the effect of the parameters used in GPVI-MFRL and the fidelity of the simulators on the number
of samples until convergence.
Variance in Learned Transition Function
To demonstrate how the variance of the predicted transition
function varies from the beginning of the experiment to convergence, we plotted heatmaps of the posterior variance for
Gazebo environment transitions. The GP prediction for a
state-action pair gives the variance v 2x and v 2y, respectively,
for the predicted state. After convergence (Figure 5), the variance along the optimal (i.e., likely) path is low, whereas the
variance for states unlikely to be on the optimal path from
start to goal remains high, since those states are explored
less often in the Gazebo environment. Hence, utilizing the

0.4
0.35

3.4

0.3
Ratio

Goal State

3.2

0.25

2.8
0.2

2.6
2.4

0.15

2.2

0.2

0.4
Variance
(a)

2
1.8

500

1.4
(a)
0.66
0.64
0.62
0.6
Wall

0.58
0.56
0.54
0.52

(b)
Figure 5. The variance plot for the Gazebo simulator after
transition-function initialization and after the algorithm has
converged. Colored regions show the respective v 2x + v 2y
values for the optimal action returned by the planner in each
state at (a) initialization and (b) after convergence.

Samples

400
Goal

Start

0.8

600

1.6

Start State

0.6

300
200
100
0

Grid World
Gazebo
0

0.2
0.4
0.6
Variance of the Noise
(b)

0.8

Figure 6. As we lower the fidelity of the grid world by adding
more noise in grid-world transitions, the agent tends to
spend more time in Gazebo. The plots show the average and
minimum-maximum error bars of five trials. (a) The ratio of
samples collected in Gazebo and total samples (y-axis) as a
function of the fidelity of the grid world. We lower the fidelity
of the grid world by increasing the variance of the simulated
transition function. (b) The number of samples collected (y-axis)
in Gazebo increases more rapidly (as demonstrated by the
diminishing vertical separation between the two plots) than
samples collected in the grid world.

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

123

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020