IEEE Robotics & Automation Magazine - June 2020 - 89

recovery strategies, which emerge from the proposed MJMPC
without the need for manual switching between strategies.
Furthermore, a comparison between humanoid and
human balancing is conducted to show the characteristics of
the learned humanoid behavior. This comparison shows
that DRL algorithms are very powerful when used to learn a
policy (e.g., balancing) within a short development and
training time that may require humans years to learn. In
contrast, to design an engineered controller from scratch
having similar performance, months or even years are
needed for developmental iterations, mainly because of the

high redundancy and diversity of control actions, which are
yet challenging to resolve in terms of physical optimality on
a high degree-of-freedom (DoF) robot. In this regard, the
learning approach is very attractive because of the significant reduction of manual effort, and the learning architecture requires only the design of input-output and rewards.
This article sheds some light on a new paradigm: the recent
high-profile successes in DRL suggest new high-quality
value in learning methods for which discovered policies can
be used as a basis to speed up development of robotic controllers (Figure 3). As an outcome of this push recovery

Performance
Maximum Performance
Development Time
per Iteration
Development
Time per Iteration
System
Definition for DRL
Framework

System Definition
Transferred From
AI Policy

Tuning and
Evaluation

Development Time
per Iteration
Development Time

System Definition and
Mathematical Modeling of
System, Sensors, and
Actuators

AI Policy
AI-Guided Controller
Classical Controller
Starting Performance
AI Policy and Controller
Starting Performance
AI-Guided Controller

Tuning and Evaluation
Tuning and
Evaluation

Reformulation

Reformulation

AI-Guided Controller
Development
Time per Iteration

AI Policy
Development
Time per Iteration

Classical Controller
Development
Time per Iteration

Figure 2. A qualitative depiction of the performance of controllers over iterations: leveraging the solution provided by an AI policy
decreases the development time of controllers compared to AI policy and traditional controller development, while starting at a
higher initial performance. The width between every black dot indicates the relative development time of the individual control design
approaches, with the shortest development time per iteration for the AI-guided controller.

Robot

Train AI
Policy
via DRL

Policy Transfer
DRL Policy

Collect Push
Recovery Data

Postprocessing
Data

Extract
Policy

Direct Deployment
Formulate
Control
Guidelines

Design
Control
Scheme

Control
Robot

Policy Transfer

Human Policy
Figure 3. The proposed control design approach of policy transfer from AI policies to real robotic platforms. By understanding the
underlying concepts of the learned policies, we can reverse engineer a controller that closely resembles the behavior of the AI policy
while being transparent and safe.

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

89
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
