IEEE Robotics & Automation Magazine - June 2020 - 101

having direct access to the policy in the form of an NN allowing
analysis of the NN in addition to merely the input-output
behavior. We performed a t-SNE analysis on the AI policy (Figure 7) and found that the DRL agent-through its interaction
with the environment-decided not to use the angular momentum modulation due to its minimal effectiveness.
Challenges and Outlook
In this article, we showed that, through an analysis of template models from human and AI policies for push recovery, a certifiably safe controller for humanoid robots can be
engineered. Currently, all analyses and implementations
were conducted in a high-fidelity physics simulator that,
despite our best efforts, differs from reality. To mitigate this
problem, we proposed a control framework for real-world
deployment that combines the proposed control law with
whole-body controllers in a cascaded manner. Due to the
shown feasibility of the motions and the reliable stability of
whole-body controllers in real-world applications, we
anticipate a seamless deployment of the controller onto a
real system.
Our future work will investigate approaches in directly
bridging this reality gap and use interactions with the real
environment to close the gap in the DRL process. Moreover,
we aim to apply this principled approach of designing controllers from template models to other systems (e.g., quadrupeds) and tasks (locomotion and manipulation).
Acknowledgments
This work was supported by the Engineering and Physical
Sciences Research Council (EPSRC) Centres for Doctoral
Training in Robotics and Autonomous Systems (EP/
L016834/1) and the EPSRC Future AI and Robotics for Space
(EP/R026092/1).
References
[1] J. Hwangbo et al., "Learning agile and dynamic motor skills for
legged robots," Sci. Robot., vol. 4, no. 26, p. eaau5872, 2019. doi: 10.1126/
scirobotics.aau5872.
[2] S. Gu, E. Holly, T. Lillicrap, and S. Levine, "Deep reinforcement
learning for robotic manipulation with asynchronous off-policy
updates," in Proc. IEEE Int. Conf. Robotics and Automation, 2017, pp.
3389-3396. doi: 10.1109/ICRA.2017.7989385.
[3] M. Zhang et al., "Deep reinforcement learning for tensegrity robot
locomotion," in Proc. 2017 IEEE Int. Conf. Robotics and Automation
(ICRA), pp. 634-641. doi: 10.1109/ICRA.2017.7989079.
[4] T. Flash and N. Hogan, "The coordination of arm movements: An
experimentally confirmed mathematical model," J. Neurosci., vol. 5, no.
7, pp. 1688-1703, 1985.
[5] C. Yang, K. Yuan, W. Merkt, T. Komura, S. Vijayakumar, and Z. Li,
"Learning whole-body motor skills for humanoids," in Proc. 2018 IEEERAS 18th Int. Conf. Humanoid Robots (Humanoids), pp. 270-276. doi:
10.1109/HUMANOIDS.2018.8625045.
[6] N. A. Radford et al., "Valkyrie: NASA's first bipedal humanoid
robot," J. Field Robot., vol. 32, no. 3, pp. 397-419, 2015. doi: 10.1002/
rob.21560.

[7] E. Coumans and Y. Bai, "Pybullet, a python module for physics simulation in robotics, games and machine learning," 2017. [Online]. Available: https://pybullet.org/wordpress
[8] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, "Trust
region policy optimization," in Proc. Int. Conf. Machine Learning, 2015,
pp. 1889-1897.
[9] D. Silver et al., "A general reinforcement learning algorithm that
masters chess, shogi, and go through self-play," Science, vol. 362, no.
6419, pp. 1140-1144, 2018. doi: 10.1126/science.aar6404.
[10] L. v. d. Maaten and G. Hinton, "Visualizing data using t-SNE," J.
Mach. Learn. Res., vol. 9, pp. 2579-2605, Nov. 2008.
[11] P. Zaytsev, W. Wolfslag, and A. Ruina, "The boundaries of walking
stability: Viability and controllability of simple models," IEEE Trans.
Robot., vol. 34, no. 2, pp. 336-352, 2018. doi: 10.1109/TRO.2017.2782818.
[12] C. McGreavy et al., "Unified push recovery fundamentals: Inspiration from human study," in Proc. 2020 IEEE Int. Conf. Robotics and
Automation (ICRA), to be published.
[13] W. Hu, I. Chatzinikolaidis, K. Yuan, and Z. Li, "Comparison study
of nonlinear optimization of step durations and foot placement for
dynamic walking," in Proc. 2018 IEEE Int. Conf. Robotics and Automation (ICRA), pp. 433-439. doi: 10.1109/ICRA.2018.8461101.
[14] S. Feng, X. Xinjilefu, W. Huang, and C. G. Atkeson, "3d walking
based on online optimization," in Proc. 2013 13th IEEE-RAS Int. Conf.
Humanoid Robots (Humanoids), pp. 21-27. doi: 10.1109/HUMANOIDS.2013.7029950.
[15] A. Escande, N. Mansard, and P.-B. Wieber, "Hierarchical quadratic programming: Fast online humanoid-robot motion generation," Int. J. Robot.
Res., vol. 33, no. 7, pp. 1006-1028, 2014. doi: 10.1177/0278364914521306.
[16] S. Kajita, H. Hirukawa, K. Harada, and K. Yokoi, Introduction to
Humanoid Robotics, vol. 101. New York: Springer-Verlag, 2014.
[17] D. Gordon, G. Henderson, and S. Vijayakumar, "Effectively quantifying the performance of lower-limb exoskeletons over a range of
walking conditions," Front. Robot. AI, vol. 5, p. 61, June 2018. doi:
10.3389/frobt.2018.00061.
[18] J. A. Nyakatura et al., "Reverse-engineering the locomotion of a
stem amniote," Nature, vol. 565, no. 7739, pp. 351-355, 2019. doi: 10.1038/
s41586-018-0851-2.
[19] H. Forssberg, "Ontogeny of human locomotor control. I. Infant stepping, supported locomotion and transition to independent locomotion,"
Exp. Brain Res., vol. 57, no. 3, pp. 480-493, 1985. doi: 10.1007/BF00237835.

Kai Yuan, School of Informatics, The University of Edinburgh,
United Kingdom. Email: kai.yuan@ed.ac.uk.
Christopher McGreavy, School of Informatics, The University of
Edinburgh, United Kingdom. Email: c.mcgreavy@ed.ac.uk.
Chuanyu Yang, School of Informatics, The University of Edinburgh, United Kingdom. Email: chuanyu.yang@ed.ac.uk.
Wouter Wolfslag, School of Informatics, The University of
Edinburgh, United Kingdom. Email: wouter.wolfslag@ed.ac.uk.
Zhibin Li, School of Informatics, The University of Edinburgh,
United Kingdom. Email: alexrobotics@gmail.com.

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

101
https://www.pybullet.org/wordpress
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
