IEEE Robotics & Automation Magazine - December 2022 - 93

serpenoid curve, as a mathematical formation of the lateral
undulation, parameterizes the angle of each body module to fit
the shape of a sinusoidal curve and drives the robot forward
using the external force generated from the side constraints.
This kinematic-based method is also extended to combine two
parameterized sinusoidal waves that propagate along the lateral
and dorsal planes for modular snake-like robots moving in
3D space [1], [3], [4]. This parameterized kinematic method,
also known as the gait equation, allows for the emergence of
complex behaviors from low-dimensional representations
with only few key parameters, greatly expanding their maneuverability
and simplifying user control. With this method,
researchers developed several biological gaits for snake-like
robots to move in indoor and outdoor environments. However,
these simplified kinematic models also pose limitations on
designing agile and energy-efficient gaits as they are confined
to those abstracted gait parameters; let alone the parameter
tuning process is inefficient and time consuming.
Designing a gait can be regarded as a parameter optimization
problem, which aims at maximizing the objective functions,
such as velocity and power. For snake-like robots, it
becomes even more challenging, and the gait equation method
mainly suffers for two reasons [5]. The extrinsic challenge
comes from the complex dynamic interaction between the
ground and the redundant mechanism with many DoF. The
intrinsic challenge is how to synchronize and coordinate all
the body joints to exhibit a proper motion pattern integrally,
which is expected to be both robust and efficient.
Conventional control algorithms for snake-like robots
require not only complex manual engineering to carefully
tweak the analytical models of the robot but also prior knowledge
of the gaits for a specific robotic system. From a control
perspective, conventional control methods often require accurate
models of the system and expertise on other domain
knowledge about the environment, which is usually complex,
inaccurate, and difficult to be acquired in the real world. From
an implementation perspective, specialized methods designed
to control snake-like robots usually comprise complex software
architectures, which are lengthy and error prone. Therefore,
it is promising to develop end-to-end learning methods
that can greatly automate the gait-design process. Such end-toend
learning methods should be able to remove the need for
accurate dynamic models or domain knowledge and can be
applied to robotic systems without explicit system identification
or too much manual engineering.
Instead of solving robotic tasks based on controllable kinematic
or dynamic models, deep RL methods solve similar
tasks by mimicking the natural learning process in the trialand-error
paradigm. Agents can learn to master diverse and
imaginative motor skills by only trying to maximize the
reward signals. Although some robots can be directly trained
in the real world to learn motor skills, it is more feasible to
learn complex skills from the simulation first as it can greatly
assist the development of RL algorithms and leads to a fast,
cheap, and safe approach by alleviating the need for expensive
and tedious real-world experiments.
On the basis of our previous work [6], this article aims to
explore the gait-generation approach by learning a neural network
(NN) controller via RL for snake-like robots and transfer
learned knowledge from simulations to real-world
implementations. We largely extend our work from [6] and
study our research in five steps. First, instead of using a simplified
snake robot, we design a new snake-robot prototype,
both in the simulation environment and real world. Second,
we design a reward signal that aims at encouraging energyefficient
locomotion. This reward function takes both the
velocity and power consumption into consideration and integrates
them after normalization. Third, instead of training
only the controller using the proximal policy optimization
(PPO) algorithm, we redesign the training pipeline by utilizing
a dynamic randomization strategy, an informative prediction
network, and the PPO algorithm to train the NN-based
controller. The randomization includes the observation space
and some of the crucial physical parameters. And the informative
prediction network can predict unobservable features
using the observable information via supervised learning.
Fourth, to show the effectiveness of the proposed method in a
more general context, we perform simulation tasks with different
robot designs (joint number) and physical parameters
(friction coefficient). We demonstrate that the proposed
method can successfully leverage the dynamics of the robot in
a general context and generate energy-efficient gaits in diverse
scenarios. Finally, we compare gait results from the NN-based
controller with results from the model-based method, both in
simulation and real-world experiments.
Our main contributions are summarized as follows. First,
different from prior works that used RL-based methods to
adapt the parameters of a low-level gait generator, our NNbased
controller takes full advantage of the flexible body of
the snake-like robot and learns sophisticated gaits, simply
from scratch. With the help of the reward signal and the iterative
RL scheme, this method offers a new alternative for
snake-like robots to study complicated motor skills without
requiring any prior knowledge. Second, with the dynamic
randomization strategy and informative prediction network,
our NN-based controller successfully learns robust and adaptive
behaviors in both simulation and the real world, where
some physical parameters and sensory measurements are
either difficult to obtain or inaccurate. Existing solutions that
are used for sim-to-real transfer, such as domain randomization
[7], can improve only the controller's performance on the
basis of a known and yet inaccurate domain. Our informative
prediction network can predict observations that cannot be
measured in a self-supervised manner. Third, by comparing
the results, we demonstrate the superiority of our RL-based
method against model-based ones in terms of energy efficiency.
Related Work
Gait Optimization With Model-Based Methods
To mimic the body structure and motion functions of biological
snakes, snake-like robots are usually designed as a chain of
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
93
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
