IEEE Robotics & Automation Magazine - June 2020 - 88

requiring only computational power. In particular, as shown
in Figure 1, our AI policy exhibits human-like behaviors with
four typical push recovery modes emerging naturally: ankle,
hip, toe, and stepping strategies.
Although the learned control policy could possibly be
deployed on a real robotic system, the lack of explainability and
analytical reasoning of the neural network (NN) makes such a
policy unsuitable for safety-critical applications in the real
world. Furthermore, due to the demand of large data and the
sample-inefficient nature of DRL algorithms, complex policies
are typically trained in simulation, which cannot guarantee the
same performance when transferred directly to the real system
[1], and the challenge of the reality gap raises concerns about
safety and performance.
To benefit from both the safety and interpretability of a control policy and the versatility and adaptability gained from learning, we propose to take advantage of DRL to quickly discover
versatile, deployable policies and solutions for very difficult problems and then study, analyze, and extract the principles of those
policies as guidelines for developing engineered controllers in a
reliable manner. By doing so, we utilize artificial intelligence (AI)
solutions for rapid control development (Figure 2) to design safe
and certifiable controllers, which can be verified and deployed on
real-world robots (Figure 3).
Although classical control development is based on gradually building knowledge that increases performance incrementally, using a template policy provides disruptive, innovative
solutions that escalate performance (green line, Figure 2). DRL
is able to achieve good performance by a number of iterations
in the DRL learning framework. However, the achieved performance is still comparatively low compared to what tuning

(a)

(b)

in control can do. Combining both approaches to kick-start
the iteration process helps in designing good controllers. After
determining the system and controller, it is a straightforward
approach to improve upon these because we are able to understand why the performance is lower than optimum; in the case
of DRL, on the other hand, there is little influence from
human engineers to improve performance, aside from reshaping the reward and/or altering the learning framework and
relying on the exploration being sufficiently large to achieve
high performance.
In this article, we study a viable approach to infer the
underlying principles of an AI policy by studying its perception-action relation, i.e., to some extent, reverse engineering
an equivalent controller in terms of functionality based on a
black box policy. This methodology is applicable not only to
AI policies but also to any black box policies, such as a human
policy. Without knowing exactly how push recovery policies
are realized by artificial NNs or biological human NNs, we
can still analyze their behavior at the functionality level by
studying their input-output relationship.
Based on evidence of optimality in human manipulation
tasks [4], we hypothesize that policies for push recovery in
humans and humanoids are optimal control processes that
follow certain optimal criteria, which can be quantified. Following this hypothesis, we analyze and utilize input-output data
collected from both humanoid and human policies and propose a minimum-jerk model-predictive control (MJMPC)
framework able to quantitatively reflect both AI and human
push recovery policies. The engineered controller has high similarity (a coefficient of determination more than 90%) with the
collected data and also exhibits the same human-like push

(c)

(d)

Figure 1. The human-like push recovery strategies emerging from DRL. The discovered behaviors serve as a guideline for the design of
certifiable and safe controllers that replicate advantageous strategies from artificial intelligence (AI) policies. The (a) ankle, (b) hip, (c)
toe, and (d) stepping strategies.

88

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
