IEEE Robotics & Automation Magazine - December 2022 - 97

1
rt =1
.
v
() cmt -
vv1
a1
.
a2
.
(3)
a 021 = influences the spread of the reward curve by defining
the x-axis intersections with xv .at= ! 1 a 022 = affects
the changes of the curve's gradient. If -= the
velocity reward rv
t
1
00
vv .,
reaches the maximum value of one.
Second, the normalized value of the total power usage Pt
r ,P which is
t
rt rP1max
P ()=- .
b1
Here, rmax is the maximum reward value, and b 061
.
-2
(4)
= is the
slope of the curve. Power efficiency is influenced by the
desired target velocity. Thus, the normalized rmax
this influence by limiting the maximum value of r .P
Finally, rewards from the velocity rv
ciency rP
are combined to form the overall reward r:
()
1
rt cm t b -2
a2
=11
P
t
vv
a1
-
1
-
.
This equation replaces the rmax
(5)
in (4) with r .v With that, the
maximal power efficiency depends on the absolute value of
the difference between the desired velocity and the robot
velocity and total power consumption.
Network Architecture
Given the input (observation )oi
t
and output (action ),ai
t
now elaborate the policy network mapping oi
t
the same dimension as the action space .oi
t
to .ai
t
we
We
design a fully connected two-hidden-layer NN as a nonlinear
function approximator to the policy
r .i The input layer has
Both hidden layers
each have 200 neurons and are each followed by a rectified
linear unit layer. The final layer outputs joint position commands
for the robot. To train the network, the PPO algorithm
adapted from [23] is used.
We train our policy network on a computer with an
i7-7700 CPU and a Nvidia GTX 1080 GPU. A total of six million
time steps (roughly 7,500 episodes) are used for training.
The maximum number of time steps in an episode is 2,000.
With the environment settings of 50 ms per time step, the
training takes approximately 42 h in total simulation time and
2 h in wall-clock time for the policy to converge.
Sim-to-Real Transfer
We present two approaches that are designed for robust transfer
of the policy learned in simulation to the real world, namely,
the parameter randomization and informative network.
Parameter Randomization
Parameter randomization is an effective approach to im -
prove the robustness of the system, especially taking the
gap between simulation and real-world experiments into
Compact Observation Space
In the real world, some of the observations are difficult to
acquire, such as the joint velocity
.
zj and the joint torque .jx
We can calculate the joint velocity by differentiating two consecutive
joint positions at each time step. However, this is
infeasible due to the noise of the joint position. To solve this
issue, we also design a compact observation space that contains
only the joint position jz and the target velocity
v .t The
informative network is used to infer the corresponding joint
velocity ,
.
zj
joint torque ,jx and head velocity v1
according to
Table 3. Technical details of the robot module.
During simulations, these physical parameters
are randomly selected from the randomization
range of the baseline value of each parameter.
Parameters
Ground friction
Mass
Baselines Unit
0.6
-
0.206
Armature inertia 0.01
Motor damping 0.3
kg
kg · m2
N · s/m
Randomization
Ranges
90 ~ 100%
90 ~ 100%
80 ~ 120%
90 ~ 110%
Informative Prediction Network
The observation space used in the simulation is only partially
observable or measurable in the real world. To solve this
problem, we propose an informative prediction network to
predict the joint torque, joint velocity, and head velocity
under the framework of supervised learning. We present
three steps to construct such an informative prediction network
(see Figure 2).
represents
and the power effiin
the " Energy-Efficiency Matrix " section is used to determine
the power efficiency reward component
represented by
consideration. By randomizing the physical parameters and
observations during training, the learned policy can be more
robust for deployment in real-world experiments.
Physical Parameter Randomization in Simulation
The physical parameters given in the " Robot and Model " section
are either measured or estimated, which may lead to
inaccuracy. And different settings of physical parameters will
directly impact performance of the generated gaits. When the
agent is trained in a stable environment with fixed dynamic
parameters, it usually leads to an overfitting controller, which
will not work properly in the real-world environment. We
randomly sample physical parameters, as listed in Table 3.
Observation-Space Randomization
In a real-world setup, the joint angles cannot be perfectly
measured due to several uncertain factors, such as the noise of
the encoder and the data errors or delay caused by the communication.
To eliminate the impact of this inaccuracy, we
add a Gaussian noise N(, )nv on every joint-angle position
in the observation space during training, where
n 0rad=
and v = 005rad
..
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
97
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
