IEEE Robotics & Automation Magazine - June 2020 - 98

Table 2. The average coefficient of determination (R2) over 2,000 trials between DRL and robot
trajectories for the ankle, hip, toe, and stepping
strategies.
Mean R 2
Axis

Ankle

Hip

Toe

Step

Total

0.97

0.94

0.93

0.96

0.95

0.9

0.91

0.96

0.88

0.91

height. This is possible if the gait is not performed with
stretched knees as is mostly the case in robot control tasks
that aim to prevent singularities in a stretched knee pose.

(a)

(b)

Figure 10. The system models used for control design. (a) A
point-mass model with a free-moving CoM independent of
actuation. This model is used for our proposed MJMPC. (b) The
IP model with actuated ankle torque and kick force and freemoving CoM according to the IP motions. This model is used to
validate and verify that the controller generated a feasible and
implementable trajectory.

tf (s)

0.6
0.4
0.2
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Displacement (cm)

Real Data
Piecewise Linear Approximation
Quadratic Approximation

Figure 11. The CoM displacement from a nominal pose over tf
for the robot.

produce a ground contact force of fmax = 2, 500 N and
torque of x max = 1, 000 Nm. Using the IP model, the required
ground contact forces and torques for tracking the CoM
reference can be calculated and are shown in Figure 8(a)
and (b). For all four push recovery scenarios, the required
force and torques are well below the maximal generatable
ground contact force and torque of fmax = 2, 500 N and
x max = 1, 000 Nm, respectively.
In summary, feasible and stable motions are enabled using
this proposed control framework to first generate push recovery motions via MJMPC and then track the motions using a
whole-body controller. The inequality constraint (5) of the
MJMPC optimization problem explicitly considers at which
speed the whole-body controller can track a reference
motion; therefore, the trackability of the reference motion is
guaranteed. Furthermore, for the whole-body controller, the
CoM height modulation does not necessarily come from a
toe lift but can be any leg extension that increases the CoM
98

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

Benchmarking Between Human and AI Policies
To this point, we have shown that methods of policy extraction can be used to estimate and reconstruct control policies
from DRL. Interestingly, when these methods are applied to
human movements, similar strategies are revealed [12]. In
this section, we compare the findings from our previous work
with those based on the AI policy to highlight the similarities
between policies extracted from humans and DRL. This will
demonstrate that, even though training in DRL can take just
hours, the emergent behaviors are similar to human behaviors, which are highly successful only after being refined over
months. These similarities show that DRL can be a high-quality source of inspiration for control design because it can
closely match human-level behavior.
Data Collection
To compare human and DRL push recovery policies, human
data was recorded in 60 trials while short impulses were
applied close to their CoM [Figure 13(a)], analogous to those
applied to robots during DRL testing. After the push ends
[Figure 13(b)], the subject takes a recovery action [e.g., stepping in Figure 13(c)]. Different impulse magnitudes were
applied to obtain a wide range of push recovery behavior.
Impulses were measured by a force/torque sensor. Movement
was measured via a Vicon motion-tracking system, and the
data were postprocessed using OpenSim and gait analysis
tools, as discussed in [17].
Push Recovery Strategies in Humans
and AI Policies
An analysis of the human data [12] shows that various strategies are used for push recovery, as depicted in Figure 1. Each
strategy is associated with a control action such as the modulation of CoP, angular momentum, CoM height, or support
polygon. Interestingly, a very similar structure also emerges in
the AI policy, which demonstrates the similarities between
DRL and human policies.
Control Strategy
We find that a single controller implementing the MJMPC
scheme in the "Understanding and Learning the Fundamental

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020