IEEE Robotics & Automation Magazine - December 2022 - 94

body modules. These modules are used to interact with the
environment to propel the robot, thereby resulting in a highly
complex system with which to be controlled. A biomorphic
approach, i.e., a gait equation [1], [2], is widely used to control
the locomotion of snake-like robots as it can easily mimic
snake-like motions by shaping its body as a sinusoid curve.
The gait equation can be regarded as a process to simplify
parametric representations of snake-like trajectories. Crespi
and Ijspeert adopted a heuristic optimization algorithm to
rapidly adjust the travel speed of an amphibious snake robot
[8]. Tesch et al. used the Bayesian optimization approach to
regulate open-loop gait parameters for snake robots, which
made the robot move faster and more reliably [9]. However,
all these works were confined by the parameterized gait-generation
system and have a very limited effect on further
improving the energy efficiency of a gait. Moreover, this
open-loop method is also time consuming and inefficient
[10] for fine-tuning gait parameters to achieve expected control
objectives.
Gait Optimization With RL-Based Methods
As an intelligent trial-and-error learning method, RL brings
new solutions for free gait-generation tasks without knowing
precise models or prior knowledge. RL technologies were
used to fine-tune parameters for motor skills in a modelbased
manner, which can either be the controllable model of
the system [11] or another abstracted gait generator such as
the central pattern generator (CPG) [12]. Shi et al. [13] developed
a simplified snake-like robot with two actuated joints
and implemented a controller based on a deep Q-network
algorithm. They demonstrated that their method could produce
meaningful gaits. Similar to the concept in [14], Liu et al.
proposed an RL-based controller to modulate the activity of a
CPG controller for generating goal-reaching and tracking
behaviors for a soft, snake-like robot in simulations [15].
Chatterjee et al. proposed a policy-improvement method to
choose different control parameters for a snake-like robot
with screw-drive units [16]. Sartoretti et al. presented an
RL-based approach to adapt the shape parameters of a snakelike
robot in response to external sensing [17].
Sim-to-Real Transfer
To narrow simulation-reality gaps, researchers have attempted
to improve simulation fidelity by building up accurate models
and refine them using real data. For instance, to match the
performance of the real system, an actuator can be modeled
and refined using its real data [18]. Researchers have also
worked on improving the robustness of a learned policy to
variations of system properties and perception information,
thereby enabling it to be feasible and adaptive in real-world
systems [19]. The task to transfer a policy learned from simulation
to the real world is treated as an instance of domain
adaption, where the source domain (simulation) is modeled
as close as possible to the target domain (real world) [7].
Robotic arms are the most common agents to deploy learned
policy from simulations as the kinematics and dynamics can
94 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
be accurately modeled. Christiano et al. controlled a robotic
arm using learned policy from simulation, of which the
inverse dynamics (from observations to actions) of that robot
was learned from the data generated by the forward dynamics
(from actions to observations) in simulations [20].
Domain randomization is another technique for sim-toreal
transfer based on the introduction of higher variance in
the domain-specific-but task-irrelevant-features of the
training data. Peng et al. propose introducing variability in the
dynamics of the simulation by sampling values for strategic
features (e.g., friction, mass, and damping coefficients) during
the training phase [21]. They argued that although modeling
the simulation in accordance with the real environment is
important for the sim-to-real policy transfer, a complete and
accurate model is often impossible due to many reasons, such
as unforeseen forces, left-out environmental characteristics,
calibration issues, and so on. In the meantime, Wulfmeier et al.
argued that an unfavorable domain in randomization can lead
to poor performance of the policy that is transferred from simulation
[22]. They proposed another idea to tackle systematic
model discrepancies: aligning the distributions over visited
states between the simulated and the real-world agent. They
demonstrated their argument by performing experiments
between two simulations with either different parameterizations
or completely different simulation engines to create situations
of misaligned and unknown system dynamics.
Robot and Model
Snake-Like Robot
Our planar snake-like robot adopts a modularized design
manner, which consists of eight identical actuated body modules
and one head module to slither forward (see Figure 1).
Each module is connected to the adjacent module by an actuated
joint, which can rotate 90° in both the left and right
directions. Each body module is composed of an actuation
system, the control system, housing components, and a pair
of passive wheels to imitate the anisotropic friction property
of the snake skin. The actuation system consists of a servo,
gearbox, and an angular sensor to feed back the angular position
of the output joint. The dc servo has a maximum torque
of 12.8 Kg cm and drives a gearbox with a reduction factor of
3.71. The angular sensor is a Hall effect encoder that is used
to measure the angular position of the output shaft of each
module. The control system is a customized STM32 microcontroller.
The STM32 runs three tasks: controlling the servo,
reading the joint angle, and communicating with the other
modules. A real-time operation system is executed on the
control system, and the control loop is set as 20 Hz. Table 1
summarizes the technical specifications of the robot.
Simulation Model
We model the snake-like robot as close as the real-world
counterpart and simulate it in Multi-Joint dynamics with
Contact (MuJoCo). As the fundamental component of a
motion's behavior, the dynamic model plays a great role in
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
