IEEE Robotics & Automation Magazine - June 2020 - 125

π radian
8

(a)

(b)

(c)

Figure 9. We used (a) the Python-based simulator Pygame as R 1, (b) Gazebo as R 2, and (c) the Pioneer P3-DX robot in the real
world as R 3 .

Average Cumulative Reward in the Real World
In Figure 10, we compare the GPQ-MFRL algorithm with
three other baseline strategies by plotting the average
cumulative reward collected by the robot as a function of
samples collected in the real world. The three baseline
strategies are
● directly collecting samples in the real world without the
simulators (direct policy)
● acquiring 100 samples in one simulator and transferring
the policy to the Pioneer robot, with no further learning in
the real world (frozen policy)
● attaining 100 samples in one simulator and transferring the
policy to the robot while continuing to learn in the real
world (transferred policy).
We observe that the direct policy performs worst in the
beginning, which can be attributed to the fact that the robot
started to learn from scratch. The frozen policy starts off better since it has already learned a policy in the simulator. However, it tends toward a lower value of average cumulative
reward, which suggests that the optimal policy learned in the
simulator is not the optimal policy in the real world.
Although the transferred policy seems to perform better at
the beginning than the frozen policy, it is difficult to state
definitively that this will always be the case.
The direct policy has a large performance variance in the
beginning. GPQ-MFRL outperforms the other strategies
right from the beginning; we attribute this to the fact that
the GPQ-MFRL collects more samples from the simulator

in the beginning and, hence, starts better from the outset. If
the transferred policy and frozen policy had been allowed
to collect more samples from the simulator, they might
have performed the same as the GPQ-MFRL. However,
deciding how many samples one should allow is a nontrivial task and problem specific. GPQ-MFRL can decide the

Table 1. Parameters used in GPQ-MFRL.
Description

Type

Value

102.74

[2.1, 5.1, 14, 6.2, 15, 2, 2, 1]

sum
v th

Confidence parameters

v th

Algorithm

Hyperparameters

20
Average Cumulative Reward
Collected in Real World

possible options: " - r/9, - r/8, f, r/9 , . The reward in each
state was set to be the sum of laser readings from seven directions except when the robot hits the obstacle. In case of a collision, it gets a reward of −50.
We trained the GP regression: Q (s, a) : R 8 " R . Hyperparameters of the squared-exponential kernel were calculated
offline by minimizing the negative log marginal likelihood of
2,000 training points, which were collected by letting the
robot run in the real world directly. The parameter values for
experiments in this section are given in Table 1.

19
18

Frozen Policy
Transferred
GPQ-MFRL
Direct

17
16
15
14
13

90 180 270 360 450 540
Samples Collected in Real World

630

Figure 10. The average cumulative reward collected by the
Pioneer robot in the real-world environment as a function of the
samples collected in the real world. The plot shows the averages
and standard deviations of five trials.

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

125

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020