IEEE Robotics & Automation Magazine - June 2023 - 90

of tumbling motion. With too low of a rate, simulated joint
control was inconsistent at low velocities. The rate, number
of simulation steps per episode step, and number of steps per
episode were empirically chosen to result
in fast training times, a policy that converges,
and consistent simulator behavior.
Measurements of the physical prototype
were translated into simulation through
URDF models, similar to the process
described in [9]. We segmented the structure
of the robot into three parts: the body and
two legs. For modeling inertial characteristics,
calculations were done assuming uniform
density, using the measured mass and
dimensions. A high friction coefficient, 0.9
for lateral friction, was chosen empirically
to minimize behavior such as sliding that
would be unlikely to transfer well [10]. As
described in the " Hardware " section, these
values were used as the means for randomized
generated URDF models. This randomization
affects the dynamics of the tumbling
robot's movement.
The tumbling bot's servo controllers do not provide consisthe
LSTM achieved slightly higher rewards at times. We
also tested different RL algorithms,
"
including Advantage
IT IS OFTEN THE CASE
THAT REAL-WORLD ENVIRONMENTS
INCLUDE
COMPLEX TERRAINS
COMPOSED OF VARYING
SURFACES, RAPID
CHANGES IN ELEVATION,
AND IMPASSABLE
OBSTACLES.
„
tent torque. At near-zero velocities, the torque is significantly
reduced. This behavior is approximated in simulation by setting
the motor's force to 0.1 N with Gaussian noise, where
the standard deviation is 0.1 N. At higher velocities, the servo
torque is set to 10 N. The servo velocity was also randomized,
with a standard deviation of 0.8 rad/s. This randomization is
significant relative to the commanded velocities, but it was
essential to having a robust policy.
RL
We used the Stable Baselines implementation of PPO with a
multilayer perceptron (MLP) LSTM policy. We set the
learning rate to 3e-4, the same that was used in [7]. Nminibatches
was set to one. For the hyperparameter selection, we
used the same ones proposed in [12], due to high performance
in their testing/evaluation. A selection of values is
available in Table 1.
Table 2 shows initial results that support our choice of a
recurrent policy. Both policies performed similarly, although
TABLE 1. The PPO2 hyperparameters used in training.
PARAMETER
Learning rate
Episode length
Batch size
Discount (c)
Clipping parameter
Optimizer
VALUE
3e-4
20 steps
128 steps
0.99
0.2
Adam
Actor Critic (A2C) [19] and Trust Region Policy Optimization
(TRPO) [20]. These are the only other
algorithms within Stable Baselines that are
compatible with the MultiDiscrete action
space, so testing other algorithms would
require modifying the tumble bot action
space. In Table 2, we see that A2C achieved
a large asymptotic reward, which was
similar to PPO-LSTM. On the other hand,
TRPO did not perform that well, followed
by PPO-MLP. Each algorithm was able to
train a successful policy, which was expected
due to the small action space, but the
higher reward values indicate a policy that
can more quickly and accurately reach the
target. Since each RL algorithm we tested
performed equally well or worse than PPOLSTM,
we chose to apply this method for
our task; in general, it is also the most stable
and likely to converge [12].
PPO allows discrete and continuous action and observation
spaces. A discrete action space was used, with three
possible outputs for each of the two legs. These outputs are
velocities of -5.25 rad/s, 0 rad/s, and 5.25 rad/s. This small
action space allows for a simple approximation of the realworld
servos in simulation. Additionally, limiting the number
of actions greatly limits the dimensionality of the policy
optimization. Adding the full range of servo velocities would
exponentially increase the amount of exploration needed
during training and would not add significant physical
capability to the robot agent.
The observation space is continuous and includes the position
of the center of mass, orientation of the robot torso, and
the intended output velocities. The center of mass velocity is
included in the reward calculation but not in the observation.
This way, after transfer, the learned policy can still be loaded,
and the velocity data do not need to be estimated online.
Decreasing the dimension of the observation space was found
to improve transfer success in [8].
Adding noise to the output velocities, as described in
the " Simulation Environment " section, prevents the policy
TABLE 2. The asymptotic reward of different
RL methods.
ALGORITHM
PPO-LSTM
PPO-MLP
A2C
TRPO
REWARD
9.342
7.758
9.333
8.089
A2C: Advantage Actor Critic; TRPO: Trust Region Policy Optimization.
90 IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
