IEEE Robotics & Automation Magazine - June 2023 - 88

Developing a control policy for tumbling robots can be a
notoriously complex task, especially in diverse environments.
Usually, these methods utilize feedback from the robot's motors
and make assumptions about the environment's characteristics,
such as specific surface friction and zero elevation gain. Previous
work has focused on motion primitives for locomotion on
a flat surface [1], while there have also been discussions for
climbing step locomotion in [3] and [6], though neither of those
demonstrated a control policy for tumbling locomotion. The
movement of these robots is highly dependent
on environmental conditions, with
varying friction, terrain slope, and uneven
surfaces limiting its performance. An adaptable
and robust control policy that can leverage
complex maneuvers and handle diverse
terrain would greatly expand the capabilities
of tumbling robots.
RL has proved to be a successful conoffer
better sample efficiency, the improved stability of onpolicy
algorithms, along with the relatively inexpensive sampling
process of training within a simulated environment,
makes PPO a more attractive method.
Usually, the policy network of PPO is a fully connected neural
"
trol technique when we lack a full model of
a robot's locomotion; furthermore, the use
of deep RL to train a model in simulation
and then transfer it to real-life control has
been demonstrated in many different contexts
[7]-[10]. The real-world environment
can never be perfectly replicated in simulation, so simulation
parameters, such as friction, gravity, and motor responses, are
varied through a process called domain randomization. This
results in a network more capable of generalizing to different
environments, thereby making it more robust to transfer from
simulation to the real world [9]. With sufficient domain randomization,
the real world is a subset of the simulated environments
used for training. A prominent example where this methodology
achieved success is [7], where a large set of randomized
environments was used in simulation-to-real transfer.
Similarly, Maekawa et al. attached irregularly shaped
branches to servos and implemented a simulation-to-real locomotion
controller [10]. This demonstrates the use of RL in
developing control policies for nontraditional robot designs.
However, in that work, tumbling motions were avoided, as the
robot was tethered. Tan et al. developed a locomotion policy for
a quadruped [8]. Tumbling robot motions were generated in [11]
but tested with a simple robot in simulation and applied only
to movement in a 2D plane. The first time that RL was used to
control a physical tumbling robot was in [2], and in this article,
we explore how well RL control policies generalize to new and
complex real-world environments.
RL METHOD AND ROBOT DESIGN
Proximal Policy Optimization (PPO) [12] is an on-policy RL
algorithm that belongs to the family of policy gradient methods
[13]. In [12], PPO establishes its importance over other
methods, such as actor-critic approaches [14], due to algorithmic
simplicity and better convergence rates. Many recent
works have demonstrated success using this method for robot
control applications, including the control of a humanoid
robot [15]. Although methods, such as soft actor-critic, can
88 IEEE ROBOTICS & AUTOMATION MAGAZINE JUNE 2023
„
EXISTING CONTROL
METHODS APPLY ONLY
A SUBSET OF POSSIBLE
ROBOT MOTIONS AND
MAKE THE ASSUMPTION
OF FLAT TERRAIN.
network, but recurrent neural network policies can also be used
in RL algorithms to improve robustness when using noisy data
[16]. Recurrent neural networks allow the policy to observe its
past actions, which can lead to more sophisticated behavior, such
as attempting a different action if the first was
unsuccessful. The authors of [7] applied this
concept for the task of manipulating cubes
with a robotic hand. In the aforementioned
application, PPO was successfully used to
optimize a recurrent neural network; in particular,
long short-term memory (LSTM)
[17] was used. For our application, we chose
an LSTM policy for to its robustness to noise
[7], [16], and we trained the policy with PPO
because of its fast convergence and effectiveness
in continuous state spaces [12].
We designed a new tumbling robot to
serve as a testing platform. Like the one in
[2], this was inspired by the Adelopod [18],
and it uses two 1-DoF legs attached to continuous rotation
servos. Similarly, no servo position feedback was utilized,
and encoders were not added to provide it. The new robot
features higher body friction, which improves its ability to
overcome uneven terrain and obstacles. Robot Operating
System (ROS) was used onboard to allow modular configuration.
After training the policies to convergence in simulation,
we evaluated their performance in various real-world
environments as well as in simulation. Our results demonstrate
a successful simulation-to-real pipeline for tumbling
locomotion and a robust control policy able to adapt to different
terrain environments.
APPROACH
ROBOTIC PLATFORM
HARDWARE
First, we constructed a simple and low-cost tumbling robot platform.
An image of the prototype appears in Figure 3(b), and its
simulation version is in Figure 3(a). In [3], low leg friction with
high body friction was found to increase performance in step
climbing. Accordingly, the legs were cut from low-friction Delrin
plastic, and the exterior aluminum standoffs were coated in
rubber to increase body friction. Two Power HD continuous
rotation servos were used for actuation. These servos are inexpensive
yet provide sufficient torque and velocity at a low mass.
An inertial measurement unit (IMU) was included; however, its
data were not used. Rather, we used a Vicon motion capture
system to obtain the pose and ensure accurate position feedback
to the robot. Since the purpose of the IMU is to enable tumbling
locomotion outside of the laboratory setting, the use of filtered
IEEE Robotics & Automation Magazine - June 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2023
