IEEE Robotics & Automation Magazine - June 2020 - 100

Table 3. The coefficient of determination (R2)
between human and robot trajectories for the
ankle, hip, toe, and stepping strategies.
Mean R 2
Axis

Ankle

Hip

Toe Lift

Step

Total

0.76

0.84

0.95

0.89

0.86

0.95

0.92

0.86

0.59

0.83

required approximately two weeks of work, including setting
up experiments, collecting subjects, carrying out experiments, and processing data. These labor-intensive logistics
further motivate the use of DRL to transfer policies over the
collection of human data because a lot more data can be
extracted in a shorter time from DRL trials.
2) Policy transfer to nonhumanoid robots: Using humans as a
template policy allows a policy transfer only to humanoids.
In contrast, DRL policies can be applied to systems that are
nonhumanoid, such as quadrupeds or multilegged robots,
or where the template is not available, e.g., extinct vertebrates [18].
3) Analysis of the internal mechanisms of the policy: For an
analysis of the policy beyond its input-output relations,
e.g., t-SNE analysis (see the "Analyzing the AI Policy" section), the AI policy is more accessible than that of the
human policy. The AI policy is represented as an NN, and
the analysis tools outlined in the "Analyzing the AI Policy" section can be leveraged to gain insights into the
mechanisms of the AI policy. Analyzing the mechanisms
of the human policy, on the other hand, requires a neuroscientific understanding of the brain and other involved
components of humans. The summarized strengths and
limitations of DRL in a number of different areas with
respect to classical control and human control can be
found in Table 4. Note that the limitations of DRL in optimality, robustness, and safety are canceled out by the
strengths of classical control in these areas, and vice versa.
The "Control + DRL" paradigm can overcome the difficulties encountered during human-inspired control (the
right-hand column).

Discussion and Conclusions
In this article, we presented an alternative application of DRL:
instead of directly deploying the AI policy, we aimed to formulate control guidelines from the AI policy. As a result, we
bypass the major drawbacks of learned policies-unsafety
and stability issues-and obtain a certifiably, optimal controller that demonstrates human-like behaviors (Figure 1).
These results were obtained for the challenging task of
push recovery.
Results
We showed that DRL is powerful enough to learn complex
motions that resemble those of humans. The learned policy
demonstrated the same push recovery strategies that can be
observed in humans and exhibited similar robustness as
state-of-the-art control algorithms. After analyzing the
learned AI policy, we used it as a guideline and template for
control design.
The engineered controller is able to reproduce the same
strategies as those of human and AI policies with a close
quantitative fit to the collected data. As observed in humans
[12], the policy minimizes jerk and implements feedback control via MPC. Furthermore, an analysis of the required
torques and forces on the system shows that the trajectories provided by the engineered policy are realizable on the
real system.
We further compared the decoded DRL and human push
recovery policies and found, surprisingly, that the AI policy
has a strong similarity to human policies. We hypothesized
that both the AI and humans are able to identify the key features in the problem, as required for high-quality performance. This finding is interesting due to the time required for
the DRL agents to acquire human-comparable push recovery
abilities: learning a good AI policy requires 6-8 h, and human
infants require 10-18 months to learn the ability of locomotion [19].
Even though MJMPC was able to reproduce both policies
and both human and AI policy can be used as a template model
for control design, we found the usage of AI policies for template models to be more advantageous. This was due to the DRL
data being immediately available-in contrast to the human
data, which required time-intensive postprocessing-and

Table 4. A comparison of various control paradigms.
Attribute

Control

DRL

Humans

Control + DRL

Control + Human

Optimality

Optimal

Suboptimal

Near optimal

Optimal

Robustness

High

Low

High

Behavior-emergence time

Weeks

Months

Weeks

Generalizability

High

Low

High

Low

Data collection time

N/A

Low

High

Low

High

Prior knowledge required

High

Low

High

Low

High

Safety/accountability

High

Low

High

N/A: not applicable.

100

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020