IEEE Robotics & Automation Magazine - June 2020 - 90

study, we obtain a certifiable, analyzable optimal controller
that does not require any state machine or switching mechanism, while exhibiting human-like push recovery strategies,
such as ankle, hip, toe, and stepping, all in a coherent optimization process.
Generating Complex Motions for Humanoid
Robots Through DRL
To use DRL policies as a basis for analysis, these policies
must reach a certain performance threshold that ideally surpasses traditional control approaches in both the types of
motions it can generate and the amount of disturbances it
can withstand. DRL is capable of learning locomotion and
fall recovery policies that surpass traditional control
approaches for quadruped robots in terms of power efficiency and versatility of motion [1].
In this section, we present a hierarchical learning framework for achieving versatile behaviors during push recovery for humanoid robots, as proposed in [5]. The learned
policy exhibits a wide range of balancing strategies comparable to human push recovery. In particular, the learned
policy is able to withstand external disturbances by modulating the center of pressure (CoP) (ankle strategy), angular
momentum (hip), center of mass (CoM), height (toe), and
support polygon (stepping), and the policy surpasses traditional control methods with respect to the disturbances it
can withstand.
All motions are trained for deployment on NASA's humanoid Valkyrie [6], a 44-DoF bipedal humanoid robot designed
for operating in disaster-response scenarios and extraterrestrial
planetary space missions such as unmanned predeployments
on Mars. Valkyrie is built to operate in human-engineered environments, standing 1.87-m tall and weighing 129 kg with ranges of motion similar to humans. The locomotion and
manipulation of Valkyrie is enabled through 25 series-elastic
actuators in its arms, torso, and legs; its respective joint limits
are described in Table 1. For sensing, Valkyrie has proprioceptive and exteroceptive sensors consisting of multiple gyroscopes, accelerometers, load cells, pressure sensors, sonar, lidar,
depth cameras, and stereo sensors.
For training the policy, the physics simulator PyBullet [7]
was used. PyBullet is able to simulate physics faster than
real-time expediting of the training process and also to

simulate collisions and soft and rigid dynamics by loading
objects defined in the unified robot description format
(URDF). The Valkyrie URDF model closely replicates the
real robot and is provided by NASA; it includes physical
quantities such as inertia, mass distribution, sensor noise,
friction, and damping [6].
Learning Framework
Our DRL framework (Figure 4) has an actor-critic architecture, in which both the actor r i (a | s) and the critic Vz (s) are
NNs parametrized by the weights i and z, respectively.
The AI policy r i (a | s) is trained through a policy gradient method called trust-region policy optimization (TRPO) [8].
Policy gradient methods directly model and optimize the policy that will yield the highest reward J (i):
J (i) =
=

/d

r

(s ) V r (s )

/d

r

(s)

s!S

s!S

/

a!A

r i (a | s) Q

r

(1)

(s, a),

with stationary distribution d r (s), state value function V r (s),
and action value function Q r (s, a) under policy r i . Using the
gradient d i J (i), gradient ascent is used to find an optimal
parameter set i that maximizes the reward.
Because the true value functions V r (s), Q r (s, a) are not
known, a critic Vz (s) estimating the true value function
V r (s) is trained. The critic Vz (s) is updated by minimizing
the expected loss L V (z) via gradient descent:
min
L V (z) = min
E [(Vz (s t) - G t) 2],
z
z

(2)

with value function estimation Vz (s t) and return Gt. The
k-t
return G t = R 3
rk is the discounted cumulative reward
k=tc
using the discount factor c # 1 and reward rt .
Using the value function estimation Vz (s t), the actor's
parameters i are updated by maximizing the reward (1) via
stochastic gradient ascent. Additionally, TRPO improves the
training stability through trust-region optimization-constraining the Kullback-Leibler (KL) divergence during a policy update, thus preventing large policy changes that could
cause instability-and reduces the variance of the policy gradient estimate using an advantage estimation A t :

Table 1. The mechanical specifications of Valkyrie.
Arm

Torso

Leg

Shoulder Shoulder Shoulder Elbow Torso Torso Torso Hip
Roll
Pitch
Yaw
Pitch Roll Pitch Yaw Roll

90

*

Hip
Hip
Pitch Yaw

Knee Ankle Ankle
Pitch Pitch Roll

Lower joint position [rad] −1.3

−2.9

−3.1

−2.2

−0.2

−0.1

−1.3

−0.6

−2.4

-0.4

−0.1

−0.9

−0.4

Upper joint position [rad] 1.5

2

2.2

0.1

0.3

0.7

1.2

0.5

1.6

1.1

2.1

0.7

0.4

Joint velocity [rad/s]

5.9

5.9

11.6

11.5

9

9

5.9

7

6.1

5.9

6.1

11

11

Joint torque [Nm]

190

190

65

65

150

150

190

350

350

190

350

205

205

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
