IEEE Robotics & Automation Magazine - December 2021 - 17

mean episode return starts at -0.23 and -0.44 for the two tasks
because of the negative penalty and the failure of the robots to
advance to the goal, and it converges to zero by the end of
learning for both policies. The maximum bands can attain values
of 0.26 and 0.28 episode return, respectively. This means
the robots learn to control flipper angles and linear velocity.
The episode return for Abs-Asc-3-COG is lower in the beginning
because the Absolem COG is higher, and the robot can
tip over more easily. Neither of the two learning curves reaches
the maximum return of one, as it is impossible to put the COG
closer than a certain distance to the point O (see Figure 2).
This is normal since the robots cannot be entirely parallel to
the ground while they traverse the staircase.
Figure 5(b) graphs the evolution of the COG during learning.
Both policies reduce the COG deviation at the same pace
and reach minimal values after 8,000 time steps, or 120 episodes.
The Absolem policy exhibits greater COG deviation,
equal to 0.19 m, due to the higher COG placement in the initial
configuration, where flippers are extended; still, it decreases
the initial COG by 17%. The Jaguar policy decreases the
COG deviation by 36%, to 0.075 m, which approximately represents
the COG deviation along the y-axis.
The learning process for the two robots is fairly similar.
Furthermore, the same episode return value is achieved by
the end of training. The COG curves exhibit comparable
behavior and converge at different values due to differences in
mass distributions. Those differences also change robot control
features; yet, both robots achieve the goal by minimizing
the COG deviation. Note, however, that the absolute COG
deviation difference between Absolem and Jaguar at the end
0.4
0.2
-0.2
-0.4
-0.6
1k 2k 3k 4k 5k 6k 7k 8k 9k 10k
Time Step
(a)
0.25
0.2
0.15
0.1
0.05
1k 2k 3k 4k 5k 6k 7k 8k 9k 10k
Time Step
(b)
Abs-Asc-3-COG
Jag-Asc-3-COG
Figure 5. The ascent task learning analysis. (a) Total reward and
(b) Dt safety measure evolution.
of learning in Figure 5(b) is not transposed to an equivalent
absolute difference in the total reward in Figure 5(a), thanks
to the normalization of each negative penalty. These results
strongly indicate that the framework is applicable to different
robot models for the ascent task.
Performance for Descent Tasks
Figure 6(a) presents learning curves for tasks Jag-Des-3-Ang
and Abs-Des-3-Ang. The first starts at a -0.1 episode return
and converges to 0.1. The second policy convergences similarly
but starts at a -0.3 episode return and ends at -0.1. Both platform
policies exhibit similar learning behavior, separated by
around 0.2, that can be explained by robot dynamics because
Absolem is exposed to higher pitch angular velocity. This is
seen in Figure 6(b), where the platform mean angular velocity
during an episode is greater for Absolem, due to a higher centroid
position over the surface in the resting state. The Absolem
policy decreases its angular velocity by the end of learning by
13%, while the Jaguar policy drops it by 10%. Minimal angular
velocity values are attained through 5,000 steps, but from this
moment onward, the learning curves slightly increase. This
may happen due to optimization of the traversal time, as a
robot spends less time for staircase traversal. As before, these
results suggest that the framework is effective for both robots.
KL Divergence Between Policies
Figure 7 gives KL divergences for learned controllers in the
3-DoF tasks. We omitted 5-DoF tasks since the simulated
Absolem model does not include an arm. An (, )ij th heat
map cell represents the divergence between a controller with
0.4
0.2
-0.2
-0.4
-0.6
1k 2k 3k 4k 5k 6k 7k 8k 9k 10k
Time Step
(a)
0.3
0.25
0.2
0.15
1k 2k 3k 4k 5k 6k 7k 8k 9k 10k
Time Step
(b)
Abs-Des-3-Ang
Jag-Des-3-Ang
Figure 6. The descent task learning analysis. (a) Total reward and
(b) Wt safety measure evolution.
DECEMBER 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
17
COG Deviation (m)
Reward
Angular Velocity (rad/s)
Reward
IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021
