IEEE Robotics & Automation Magazine - June 2020 - 92

position and velocity, and ground contact force to encourage the
robot to be close to a target position. Furthermore, a penalty
(negative value) is given proportional to the power consumption
rpower and if the upper body is in contact with the ground or the
foot is not in contact with the ground rcontact . For the contact
penalty, a constant value is subtracted if there was no ground
contact of the foot or upper body contact with the ground.
The high-level AI policy generates actions in the form of
target joint-angle references at a frequency of 25 Hz by forward propagating through the actor network using the current state as input. The robot performs its motions with upper
body joints locked in their nominal position. The action
space A ! R 11 thus consists of joint positions for torso
pitch, hip roll, hip pitch, knee pitch, ankle roll, and ankle
pitch. The target joint angles are given to a low-level proportional-derivative (PD) controller operating at 500 Hz to generate the joint torques that are ultimately applied to control
the robot [Figure 5(a)].
The state space S ! R 47 consists of the joint position and velocity of the actuated joints, pelvis states

(translational and angular velocity, and orientation),
CoM states (translational velocity and position in local
frame), ground contact force, torso position (in local
frame), and foot position (in local frame). The state is
sampled at a frequency of 500 Hz and filtered by a firstorder Butterworth filter with a cutoff frequency of 10 Hz.
More details regarding the formulations of the learning
framework can be found in [5].
Behaviors of the AI Policy
Training a robust push recovery policy in the presented learning framework requires 6-8 h of simulation time on a commercial desktop PC (Intel i7 6700 K, Nvidia Titan X, and
TensorFlow) and equates to 1-2 days in real time. The
learned push recovery policy demonstrates human-like push
recovery strategies such as the ankle, hip, toe, and stepping
strategies, that emerge at different levels of disturbance (Figure 1). The policy performs well in the presence of external
disturbances and large sensor noises due to the filtering and
sufficient amount of exploration.

Robot

Low-Level
Joint Controller

High-Level NN
Target
Joint Angles

.
Kp e(t) + Kd e (t )

Joint Torque
Guidelines
for Control Design

Joint States
Robot State
(a)
Choice of System Model

Point
Mass

LIPM

Robot

Control Design

Choice of
Controller Family

Whole
Body

Control
Formulation

(b)

Evaluation of
Fit/Similarity

Parameter
Tuning

If Not Good Fit

Controller Implementation
Reverse-Engineered Controller

Robot State

Step Location
CoM Reference
Nonlinear
Step Location
Optimizer

MJMPC

Whole
Body
Control

Robot
CoM State

(c)
Figure 5. The process for AI-aided control design. (a) The hierarchical control system for push recovery. The high-level NN is learned
as depicted in Figure 4. (b) The design process of the engineered controller. (c) The implementation of the designed controller into a
whole-body control framework. LIPM: linear inverted pendulum.

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020