IEEE Robotics & Automation Magazine - June 2020 - 124

experience from lower-fidelity simulator results in more
directed exploration of the higher-fidelity simulators.
Effect of Fidelity on the Number of Samples
Next, we studied the effect of varying the fidelity on the total
number of samples and the fraction of samples collected in
the Gazebo simulator. Our hypothesis was that, as the fidelity
of the first simulator decreases, the agent will need more
samples in Gazebo. To validate this hypothesis, we varied
the noise added to simulate the transitions in the grid world.
The transition model in Gazebo remained the same. The
total number of samples collected increases as we increase

0.5
0.4

Ratio

0.3
0.2
0.1
0
-0.1
0.1

0.2

0.3
σth

0.4

σthsum = 0.5

σthsum = 0.7

σthsum = 0.6

σthsum = 0.75

0.5

Figure 7. The ratio of samples collected in Gazebo to the total
samples as a function of the confidence parameter v th for four
different values of v sum
th . The figure shows the average and
standard deviation of five trials.

Initial State Value Function

15
10
5
0
RMax
GP-VI
RMax-MFRL
GP-VI-MFRL

-5
-10

0

200
400
600
Samples in Gazebo

800

Figure 8. A comparison of GP-VI-MFRL with three baseline
strategies. The y-axis shows the value function for the initial
state V (s 0) in Gazebo as a function of the number of samples
collected in Gazebo. The value function estimation for GP-VIMFRL converges most quickly.

124

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020

the noise in the grid world [Figure 6(b)]. As we do so, the
agent learns less accurate transition functions, leading to
more samples collected in Gazebo. Not only does the agent
need more samples, but the ratio of the samples collected in
Gazebo to the total number of samples also increases
[Figure 6(a)].
Effect of the Confidence Parameters
The GP-VI-MFRL algorithm uses two confidence parameters, v th and v sum
th , which quantify the variances in the transition function to switch to a lower and higher simulator,
respectively. Figure 7 shows the effect of varying the two
parameters on the ratio of the number of samples collected in
the Gazebo simulator to the total number of samples. Smaller
sum
v th and v th result in the agent collecting more samples in
the lower-fidelity simulator and may also result in slow convergence. Depending on user preference, one can choose the
values of confidence bounds from Figure 7.
Comparison With RMax MFRL
Figure 8 compares GP-VI-MFRL with three other baseline
algorithms:
● the RMax algorithm running only in Gazebo without grid
world (RMax)
● the GP-MFRL algorithm running only in Gazebo with no
grid world present (GP-VI)
● the original MFRL algorithm [2] (RMax-MFRL).
Specifically, we plot the value of the initial state, V (s 0), as a
function of the number of samples in Gazebo, i.e., R 2 . We
observe that GP-VI-MFRL uses fewer samples in Gazebo to
converge to the optimal value than the other methods.
GP-VI-MFRL performs a GP update at each time step.
This GP update grows cubically with the number of training
samples, which makes GP-VI-MFRL computationally infeasible beyond a certain number of training samples. However,
this issue can be addressed by using appropriate active-learning strategies, which select a subset of samples to retain, thereby keeping the size of the data set constant. The total
computational time for GP-VI-MFRL to perform GP updates
on collected samples accounts for approximately 10 min.
GPQ-MFRL Algorithm
We use three environments (Figure 9) to demonstrate the
GPQ-MFRL algorithm. The task for the robot is to navigate
through a given environment without crashing into the obstacles, assuming the robot has no prior information about the
environments. There is no goal state.
The robot has a laser sensor that gives distances from
obstacles along seven equally spaced directions. The angle
between two consecutive measurement directions was set to
be r/8 radians. The actual robot has a Hokuyo laser sensor
that operates in the same configuration. Distance measurements along the seven directions serve as the state in the environments. Therefore, we have a 7D continuous state space:
S ! (0, 5] 7. The linear speed of the robot was held constant
at 0.2 m/s. The robot can choose its angular velocity from 19
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
