IEEE Robotics & Automation Magazine - June 2020 - 91

max
J (i) = max
E;
i
i

r i ( a | s)

r i old (a | s)

experience during training, the policy will be able to generalize better to push disturbances during runtime.

A t (s t)E,

subject to E [D KL (r i old (· | s) || r i (· | s))] # d,

(3)

with updated policy r i (a | s), old policy r i (a | s), advantage
l
estimate A t = R 3
l = 0 (cm) (rt + V (s t + 1) - V (s t )), discount
factor c, variance/bias tradeoff parameter m ! [0, 1], KL
divergence D KL ($ < $), and trust region d.
During the training process, by sampling the action a t
from the stochastic policy r i (a t | s t) and performing this
action, the environment returns the resulting reward rt,
and the corresponding state s t + 1 . During every update step
during training, so-called state-action-reward-state (SARS)
tuples {s t, a t, rt, s t + 1} are collected in a batch, D. Upon
reaching a certain batch size N-the size of the batch is a
hyperparameter trading off the variance of the data against
the speed of training-the actor and critic are then updated
through gradient ascent and descent, respectively. Although
a stochastic policy provides exploration, applying random
forces to the pelvis increases the policy's ability to withstand
large pushes due to a larger coverage of the state space in
the stored experience data. Using the additional push

Inferring Actions From the AI Polvicy
Training in the presented learning framework and interacting
with the environment allow the AI policy to experience which
reward will be yielded to certain actions in different states.
Using these experiences in the form of SARS-tuple trajectories x = {s 1: N , a 1: N , r1: N , sl1: N } and reward maximization
through gradient ascent, the policy is incentivized to perform
actions that will lead to high-value states while avoiding lowvalue states. Consequently, the AI policy r (a | s) learns how
to optimally act in state s t by performing actions a t to maximize a human-defined reward function r.
The reward function is designed as
r = rpose + rCoM pos + rCoM vel + rGRF + rcontact + rpower ,

such that high rewards are given if the torso pose rpose, CoM
position rCoM pos, CoM velocity rCoM vel, and ground contact
force rGRF (equally distributed ground reaction forces) remain
close to the nominal values. A radial basis function
ri = exp (- a i (x target - x) 2) is used for the torso pose, CoM

Update Network Weights φ of Value
Function V (st) via Gradient Descent
Using Sampled Rewards r1:N

′ , r1:N } by Applying
Collect N Trajectories τi = {s1:N , a1:N , s1:N
Action at From Policy π (atst) for State st to Environment
Actor

150
tanh

50
tanh

Environment

25
tanh

Critic

150
50
25
ReLU ReLU ReLU

Joint
State

Torso
State
Pelvis
State
Feet
State

Joint
State

Joint Position Target

CoM
State

(4)

τi

CoM
State

V (st )

Torso
State

V (s)

Pelvis
State
Feet
State

Contact
Forces

Contact
Forces

θ
Update Network Weights θ of Policy π (atst) via
Stochastic Gradient Ascent of the
DRL Objective Using Advantage Estimate At

At

Calculate Advantage Estimates
At From Trajectories τi and
Current Value Function V (st )

Figure 4. A learning framework for the DRL of a push recovery policy. ReLU: rectified linear unit.

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

91
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
