IEEE Robotics & Automation Magazine - June 2022 - 84

them-are sorely needed. We believe that the vision-based
Furuta platform we have presented is ideal for such studies.
Not only does it capture important challenges for fast and
unstable systems, it is simple enough that development iterations
can be quickly made, and conventional controllers provide
a clear baseline for comparison and a tool to debug and
validate different parts of the learning pipeline. For example,
we often determined whether to put more effort into step A
or C by comparing the DNN-based SE coupled with an LQR
controller against the DNN-based policy with encoder readings.
We believe that the required effort to recreate the
vision-based Furuta platform is reasonable, and we look forward
to studies that reproduce, and improve on, the results
we have presented.
Acknowledgments
We thank Moritz Schneider for his contribution to the code
repository for the vision-based Furuta pendulum and his
efforts to successfully reproduce the platform and our results.
We also thank Andreas Doerr, Edgar Klenske, and Niklas
Funk for helpful early discussions on the setup of the project.
We thank Tiffany Cheng for help with the video and illustrations.
This work was funded, in part, through the Cyber Valley
Initiative. This article has supplementary downloadable material
available at https://doi.org/10.1109/MRA.2021.3129448,
provided by the authors.
References
[1] C. J. Ostafew, A. P. Schoellig, T. D. Barfoot, and J. Collier, " Learningbased
nonlinear model predictive control to improve vision-based
mobile robot path tracking, " J. Field Robot., vol. 33, no. 1, pp. 133-152,
2016, doi: 10.1002/rob.21587.
[2] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen, " Learning
hand-eye coordination for robotic grasping with deep learning and
large-scale data collection, " Int. J. Robot. Res., vol. 37, nos. 4-5, pp. 421-
436, 2018, doi: 10.1177/0278364917710318.
[3] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, " Learning
quadrupedal locomotion over challenging terrain, " Sci. Robot., vol.
5, no. 47, p. eabc5986, 2020, doi: 10.1126/scirobotics.abc5986.
[4] OpenAI et al., " Solving Rubik's cube with a robot hand, " 2019,
arXiv:1910.07113.
[5] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel, " Deep
spatial autoencoders for visuomotor learning, " in Proc. 2016 IEEE Int.
Conf. Robot. Automat. (ICRA), pp. 512-519, doi: 10.1109/ICRA.2016.7487173.
[6] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine,
" How to train your robot with deep reinforcement learning: Lessons
we have learned, " Int. J. Robot. Res., vol. 40, nos. 4-5, pp. 698-721, 2021,
doi: 10.1177/0278364920987859.
[7] Z. Xie, P. Clary, J. Dao, P. Morais, J. Hurst, and M. Panne, " Learning
locomotion skills for cassie: Iterative design and sim-to-real, " in Proc.
Conf. Robot Learn. (CoRL), vol. 100, pp. 317-329, 2020.
[8] J. W. Roberts, L. Moret, J. Zhang, and R. Tedrake, Motor Learning at
Intermediate Reynolds Number: Experiments with Policy Gradient on
the Flapping Flight of a Rigid Wing. Berlin, Heidelberg: Springer-Verlag,
2010, pp. 293-309.
[9] A. von Rohr, S. Trimpe, A. Marco, P. Fischer, and S. Palagi, " Gait
learning for soft microrobots controlled by light fields, " in Proc. IEEE
Int. Conf. Intell. Robots Syst. (IROS), 2018, pp. 6199-6206, doi: 10.1109/
IROS.2018.8594092.
[10] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl, " Learning by
cheating, " in Proc. Conf. Robot Learn. (CoRL), 2019, vol. 100, pp. 66-75.
[11] S. Srinivasan, I. Sa, A. Zyner, V. Reijgwart, M. I. Valls, and R. Sieg wart,
" End-to-end velocity estimation for autonomous racing, " IEEE Robot.
Automat. Lett., vol. 5, no. 4, pp. 6869-6875, 2020, doi: 10.1109/LRA.2020.3016929.
[12] S. Levine, C. Finn, T. Darrell, and P. Abbeel, " End-to-end training of deep
visuomotor policies, " J. Mach. Learn. Res., vol. 17, no. 1, pp. 1334-1373, 2016.
[13] D. Hoeller, L. Wellhausen, F. Farshidian, and M. Hutter, " Learning
a state representation and navigation in cluttered and dynamic environments, "
IEEE Robot. Automat. Lett., vol. 6, no. 3, pp. 5081-5088,
2021, doi: 10.1109/LRA.2021.3068639.
[14] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, " TossingBot:
Learning to throw arbitrary objects with residual physics, " IEEE
Trans. Robot., vol. 36, no. 4, pp. 1307-1319, Aug. 2020, doi: 10.1109/
TRO.2020.2988642.
[15] H. Ha and S. Song, " Flingbot: The unreasonable effectiveness of
dynamic manipulation for cloth unfolding, " 2021, arXiv:2105.03655.
[16] N. O. Lambert, D. S. Drew, J. Yaconelli, R. Calandra, S. Levine, and
K. S. J. Pister, " Low-level control of a quadrotor with deep model-based
reinforcement learning, " IEEE Robot. Automat. Lett., vol. 4, no. 4, pp.
4224-4230, 2019, doi: 10.1109/LRA.2019.2930489.
[17] J. Mattner, S. Lange, and M. Riedmiller, " Learn to swing up and balance
a real pole based on raw visual input data, " in Proc. Int. Conf. Neural
Inf. Process. (NIPS), 2012, pp. 126-133, doi: 10.1007/978-3-642-34500-5_16.
[18] E. Kaufmann, A. Loquercio, R. Ranftl, A. Dosovitskiy, V. Koltun,
and D. Scaramuzza, " Deep drone racing: Learning agile flight in
dynamic environments, " in Proc. 2nd Conf. Robot Learn. (CoRL), 2018,
vol. 87, pp. 133-145.
[19] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,
" Proximal policy optimization algorithms, " 2017, arxiv:1707.06347.
[20] L. Perez and J. Wang, " The effectiveness of data augmentation in
image classification using deep learning, " 2017, arXiv:1712.04621.
[21] S. Heim, A. Rohr, S. Trimpe, and A. Badri-Spröwitz, " A learnable safety
measure, " in Proc. 3rd Conf. Robot Learn. (CoRL), 2019, vol. 100, pp. 627-639.
[22] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas,
" Reinforcement learning with augmented data, " in Proc. Adv. Neural
Inf. Process. Syst. (NIPS), 2020, vol. 33, pp. 19,884-19,895.
Steffen Bleher, Intelligent Control Systems Group, Max Planck
Institute for Intelligent Systems, Stuttgart, 70569, Germany.
Email: steffen.bleher@gmail.com.
Steve Heim, Intelligent Control Systems Group, Max Planck Institute
for Intelligent Systems, Stuttgart, 70569, Germany. Email:
heim.steve@gmail.com.
Sebastian Trimpe, Intelligent Control Systems Group, Max
Planck Institute for Intelligent Systems, Stuttgart, 70569, Germany,
and Institute for Data Science in Mechanical Engineering,
RWTH Aachen University, Aachen, 52068, Germany.
Email: trimpe@dsme.rwth-aachen.de.
84 * IEEE ROBOTICS & AUTOMATION MAGAZINE * JUNE 2022
https://www.doi.org/10.1109/MRA.2021.3129448
IEEE Robotics & Automation Magazine - June 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2022
