IEEE Robotics & Automation Magazine - June 2020 - 94

humans to construct better solutions and improve the way
humans approach these problems: AlphaGo, a superhuman
policy that beat 18-time world champion Lee Seedol four
games to one, has subsequently been analyzed to provide
humans with insights on why and how AlphaGo won. The
release of AlphaGo Teach enabled Go players to analyze strategies and changed the way in which humans played the game:
AlphaGo Teach helped to quantify the value of starting the
game (suggesting that the current compensation of 6.5 points
for not starting puts the nonstarting player in an advantage),
redefined conventional openings, and utilized unconventional moves that went against conventional human wisdom and
were previously deemed disadvantageous.
Inspired by the performance of the AI policy, this section aims to present an approach to reverse engineer the
policy to obtain a unified controller exhibiting the same
push recovery strategies as that of the AI policy. Reverse
engineering the AI policy maintains its versatility while
enabling us to analyze and modify the controller with
respect to stability and safety.
We aim to find a controller that is able to stabilize the system and has a close similarity and fit to that of the DRL policy. To this end, the choice of system model and controller
type needs to be made. In this section, we demonstrate that
the DRL policy can be accurately replicated using point-mass
dynamics, controlled by a minimum-jerk controller. This
indicates that the NN may use a point-mass template internally and try to optimize the jerk.
The design process (Figure 5) involves three steps: acquiring state-action data from the AI policy, reverse engineering
the policy, and deploying the controller in a whole-body control framework. First, various state-action pairs are acquired
through simulation for different external disturbances. During the reverse-engineering process and based on the criterion of least square error fit, both a suitable system dynamics
representation and a controller using these system dynamics
will be determined. Lastly, in the deployment phase, the engineered controller will be used to generate step locations and
the CoM trajectory, which will then be tracked by a wholebody controller.
Analyzing the AI Policy
The actor NN can be analyzed thoroughly with the goal of
finding guidelines and quantities that enable general push
recovery. The insights gathered from analyzing the actor can
be used to improve controller design. Methods for analyzing
and interpreting the NN are mainly of a visual nature and
originate from the field of computer vision.
In this article, we visualize the NN's activation using t-distributed stochastic neighbor embedding (t-SNE) to project the
activation of NNs onto a 2D plane while preserving neighborhoods and clusters in projections [10]. t-SNE is performed on
the neuron activations to project the high-dimensional NN
activation onto a 2D space to investigate the generalization
behavior of the policy. In the final projection, data points that
represent a similar NN activation are grouped nearby with
94

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

high probability, whereas nonsimilar data points are distant
from each other. We treat all neuron activations (compare
Figure 4) during one time step as one high-dimensional data
point for the t-SNE analysis. For every time step of a trial, we
collect all neuron activations across disturbances ranging
from 0 to 210 Ns. During these disturbances, all four push
recovery strategies (Figure 1) occurred and are labeled by
color in Figure 7.
As shown in Figure 7, the distinct strategies cover separate
regions and thus partly explain the ability of the AI policy to
generalize across different disturbances, as the activation is similar for the same strategy. NN representations labeled as the same
strategy are perceptually similar and projected close to each
other. The points that represent the ankle strategy cluster well
with few outliers and are visually distinct from other strategies.
The toe and stepping strategies are also quite distinct from each
other. Furthermore, the lack of activations corresponding to the
hip strategy (blue points in Figure 7) indicates that the AI policy
does not utilize the angular momentum as much as the other
strategies and could suggest that the hip strategy should be used
less than other strategies, such as the toe strategy. The hip strategy does not cluster as clearly as the other strategies; thus, we
hypothesize that it is an artifact of other motions rather than an
intentional motion. The effectiveness of the hip strategy is further questioned in the literature [11], which should be kept in
mind when reverse engineering the AI policy.
Incorporating AI Policy-Inspired Principles
in the Control Design
The close resemblance between the DRL [dashed lines in Figure 8(a) and (b)] and human data [dashed lines in Figure 9(a)
and (b)], for which strong evidence of being minimum jerk
exists [4], [12], along with the typical smoothness characteristics of minimum-jerk trajectories motivates the design of a
minimum-jerk control scheme. Inspired by this, reverse engineering the AI policy involved investigating whether it is also
minimum jerk. For the control design process of reverse engineering the AI policy, we followed the same processes as those
in Figure 5. Our study eventually found that applying the controller proposed in our previous work [12] leads to the best-fitting results and a strong resemblance between the AI policy
and the reconstructed controller. In contrast, the subsequently
tested different models [point-mass, inverted pendulum (IP),
and whole-body models] and controllers (PD control, linear
quadratic regulators, and MPC) exhibit a worse fitting to the
data. We designed an MPC controller that uses point-mass
dynamics as in Figure 10(a) to minimize the jerk of the CoM
state for motion generation and feedback control. To prevent
generating arbitrarily large motions that violate the actuation
constraints of the robot, the CoM state is further constrained
in the MPC scheme.
The idea of MPC lies in solving an optimization problem
at every time step. Using the feedback of the current state, a
closed-loop control behavior can be achieved. The objective
2
t
t
function J = ^1 2h 8 0f ^ d 3 x (t) dt 3h dt = ^ 1 2h 8 0f u (t) 2 dt
is designed to minimize jerk xq (the input u of the system)

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020