IEEE Robotics & Automation Magazine - December 2021 - 14

between O and the projections of the COG on the x- and
y-axis, respectively.
Besides safety, we further observed that the contact of the
robot tracks with the surface cannot alone guarantee proper
traction. In reality, it is essential that sufficient force is exerted
from the robot to the ground, which is possible when the projection
of the COG Cx
is located near the perpendicular projection
of the geometrical center O on the staircase. This leads
to more uniformly distributed traction along the entire platform.
Finally, we further penalize the robot when it loses stability
and tips over, which happens when its pitch reaches /.2r
Platform Shaking Reduction
The effect of gravity on the movement of a robot switches
between ascent and descent by hindering ascent and accelerating
descent, which may endanger the robot and lead to tipovers.
Even if no accident happens, every staircase edge
traversal causes platform shaking and collisions, which influence
the robot and potentially an object that is being transported.
These events are repetitive when the robot traverses
every step, but the greatest impact can be observed at the
beginning of the traversal when the robot COG crosses the
uppermost stair edge. At that moment, the robot velocity can
be instantaneously increased, leading to a shock, slip, and,
worst, tip-over. The end of traversal is also crucial because if
the rear flippers are pushed down, the rear part of the robot
just falls from the step.
Such behaviors lead to distinctive pitch velocity peaks;
therefore, the controller should mitigate such events. Letting
Wt
denote pitch velocity and KW
rt = '
W
-
-KW
Wt
)
cient, the corresponding negative reward is
,
,
1if tip over
otherwise
the normalization coeffi(6)
This
penalty is assigned only in descent tasks and when a
robot traverses a set of stairs, namely, when the COG projection
on the steps is located between the staircase nosing limits.
Controller Learning Details
In designing a controller learned via RL, there are several
alternatives that can be considered [17]. Approximators such
as tile and coarse coding tend to suffer from the curse of
dimensionality and do not generalize well due to grain size
sensitivity. The main advantage of artificial NNs over other
function approximators is their generalization capability and
straightforward increase of depth, which has been well studied.
Another common choice is radial basis function (RBF)
NNs, which were widely applied before the emergence of
deep NNs. RBF NNs converge quickly to global optima with
fewer trials and errors; still, they have proved to be of limited
use due to the absence of tractable and stable integration into
more complex deep NN architectures.
Shallow networks with one hidden layer tend to have more
parameters to approximate the same function, in contrast to
deep networks with fewer parameters and the additional ability
to learn different representations at intermediate levels.
14 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2021
Thus, we opted for a multilayer perceptron (see Figure 1). The
first layer is composed of nine neurons on which the state
vector s is received every time step. The input is forwarded
through two dense layers to the output layer, which forms the
action vector a. Following up on the discussion in the " Control
Methods in Obstacle Negotiation " section, the SAC and
TD3 are considered more sample efficient than PPO since
they are off policy and converge faster to higher returns. Still,
we use PPO in our work to optimize the policy parameters
because it is more straightforward to implement, less sensitive
to hyperparameter changes, and has the most favorable
results in deep RL.
KL Divergence-Based Policy Comparison
We can quantitatively compare a pair of trained policies via a
commonly used probability distribution divergence measure,
namely, the KL divergence. Often, RL algorithms exploit divergence
as a metric to be optimized or a constraint of policy
updates, as, for example, in PPO, where updates of the policy
parameters are constrained through a penalty on KL divergence.
We denote the reference policy for a given task as
rq ,
and we wish to find how the policy pr diverges from qr
(assuming the same observation and action spaces). The process
of learning a policy amounts to learning the mean and
variance statistics of the underlying probability distribution
rather than the exact probability distribution for each possible
state-action pair. Using the hypothesis of a normally distributed
policy with a mean and variance that depend on the state, a
policy can be randomly sampled across the total observation
space, with actions chosen according to that policy.
Thus, once the total set of observations has been sampled,
the corresponding actions are prescribed by one of the two
policies being compared, namely, the reference policy
rq .
After having determined the state-action pairs across which
the policies will be compared, we calculate the divergence as
D (, )(,)
sa,
KL pq
rr = / pr sa log
r
r
q
p
(, )
(, )
sa
sa
.
(7)
Training a policy is always a stochastic procedure, and thus a
policy is a random variable itself, together with the number of
state-action pairs we choose for calculating divergence. We
empirically determine the number of times needed to train a
policy as well as the number of samples taken in the calculation
of the divergence, though a more consistent estimation is
possible if required.
Controller Deployment on a Real Robot
System Description
We begin by presenting the experiment setup used to obtain a
ground truth robot and environment state. This enables us to
establish an upper bound on the effectiveness of a controller
in reality, with moderate state errors, while facilitating the
development and evaluation of alternative robot localization
approaches later on. On the other hand, we opted for injecting
noise into the state during policy learning in simulation
IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021
