IEEE Robotics & Automation Magazine - June 2020 - 126

number of samples in each simulator by itself without the
need for human intervention.
Policy Variation With Time
Figure 11 shows the absolute percent change in the sum of the
value functions with respect to the last estimated sum of the
value functions and average predictive variance for states
" 1, 3, 5 ,7 in all three simulators. Initially, most of the samples
are collected in the simulator, whereas, over time, the samples
are collected mostly in the real world. The simulators help the
robot make its value estimates converge quickly, as observed
by a sharp dip in the first white region. Note that GP updates
for the ith simulator (Qt i) are made only when the robot is
running in ith simulator.
Higher-Dimensional Spaces and Sparse GPs
One of the limitations of GPs is their computational complexity, which grows cubically with the number of training samples. However, we use sparse approximations to address this
limitation. We also increase the dimensionality of the state
45
40

Percent Change

35
30
25
20
15
10
5
0

100

200

300 400 500
Total Samples
(a)

600

700

130
Pygame
Gazebo
Pioneer

Average Posterior Variance

120
110
100
90
80
70
60
50
40

100

200

300 400 500
Total Samples
(b)

600

700

Figure 11. The yellow, green, and white regions correspond to
the samples collected in the Pygame, Gazebo, and real-world
environments, respectively. Plots are for state set " 1, 3, 5 ,7.
(a) The sum of absolute change in the value functions. (b) The
average variances in value-function estimations.

126

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

space to verify whether the proposed algorithms scale to
higher dimensions. Specifically, we increase the number of
laser readings to 180 equally spaced directions. Therefore, GP
regression is used to estimate Q(s, a) : R 181 " R .
There are several methods for sparse GP approximations. We use the technique from [6] that finds a possibly
smaller set of points (called inducing points) that best fit
the data. The GP inference is conditioned on the smaller
set of inducing points rather than the full set of training
samples. Finding inducing points is closely related to finding low-rank approximations of the full GP covariance
matrix. The inducing points may or may not belong to the
actual training data.
We did several experiments with GPQ-MFRL to study
the performance of the algorithm for a number of inducing
points. We used Pyro [20] to implement sparse GPs. Figure 12 shows the average cumulative reward collected by
the robot in the Gazebo environment when GP inference is
done with the number of inducing points set to 5%, 15%,
25%, and 100% of the total training samples. We observe a
significant increase in the cumulative reward collected
when going from 5 to 15% but not much from 15 to 25%
(y-axes in Figure 12).
A plot of the wall clock times to perform GP inference in
Pygame is shown in Figure 13. The wall time to perform GP
inference in Gazebo exhibits a similar trend, which we omit
for the sake of brevity. The wall clock time represents the time
to perform all GP operations, including the time to update the
hyperparameters and find the inducing points of both GPs.
We update these after every 10 new training samples in an
individual simulator. The experiments were performed on a
machine running Ubuntu 16.04 with an Intel Core i7-5600U
CPU at 2.60 GHz, Intel HD Graphics 5500, and 16 GB of random-access memory.
The results suggest that a small number of inducing
points is sufficient and yields diminishing marginal gains in
the performance when the amount of inducing points
increases. Inference on inducing points with 25% of the
training data leads to performance almost as good as that
achieved with full GP regression, in terms of the reward collected by the learned policy [Figure 12(d)], but it is significantly faster.
Comparison Between GP-VI-MFRL and GPQ-MFRL
We compare GP-VI-MFRL and GPQ-MFRL using the
average cumulative reward collected by the robot in
Gazebo as the metric in the obstacle-avoidance task. To
do this, we used full GP regression to perform the inference. The laser obtains distance measurements from
seven equally spaced directions, and we trained seven
independent GPs to learn the transition function in GPVI-MFRL (one GP corresponding to each direction). A
performance comparison is shown in Figure 14. Although
both algorithms seem to perform the same asymptotically, GP-VI-MFRL performs slightly better than GPQ-MFRL
in the beginning.

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020