IEEE Robotics & Automation Magazine - June 2020 - 54

movements using spatiotemporal features and IIL. Challenges such as the generation of smooth, precise, and stylistic movements (i.e., dealing with high-frequency details
[20]) could be also addressed.
Acknowledgments
This research is funded by the Netherlands Organization for
Scientific Research project Cognitive Robots for Flexible
Agro-Food Technology, grant P17-01; European Research
Council Starting Grant Teaching Robots Interactively, project
reference 804907; Chile's National Fund for Scientific and
Technological Development project (FONDECYT) 1201170;
and Chile's Associative Research Program of the National
Research and Development Agency (ANID/PIA) project
AFB180004.
References
[1] S. Chernova and M. Veloso, "Interactive policy learning through
confidence-based autonomy," J. Artificial Intell. Res., vol. 34, pp. 1-25,
Jan. 2009. doi: 10.1613/jair.2584.
[2] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. Kochenderfer,
"HG-DAgger: Interactive imitation learning with human experts," in
Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019, pp. 8077-
8083. doi: 10.1109/ICRA.2019.8793698.
[3] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D.
Amodei, "Deep reinforcement learning from human preferences,"
in Proc. Advances Neural Information Processing Systems, 2017, pp.
4299-4307.
[4] W. B. Knox and P. Stone, "Interactively shaping agents via
human reinforcement: The TAMER framework," in Proc. 5th ACM
Int. Conf. Knowledge Capture, 2009, pp. 9-16. doi: 10.1145/1597735.
1597738.
[5] C. Celemin and J. Ruiz-del Solar, "An interactive framework for
learning continuous actions policies based on corrective feedback," J.
Intell. Robotic Syst., vol. 95, pp. 77-97, July 2019. doi: 10.1007/s10846-0180839-z.
[6] J. MacGlashan et al., "Interactive learning from policy-dependent
human feedback," in Proc. 34th Int. Conf. Machine Learning, 2017,
vol. 70, pp. 2285-2294.
[7] V. Mnih et al., "Human-level control through deep reinforcement
learning," Nature, vol. 518, no. 7540, pp. 529-533, 2015. doi: 10.1038/
nature14236.
[8] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997. doi: 10.1162/neco.
1997.9.8.1735.
[9] W. Böhmer, J. T. Springenberg, J. Boedecker, M. Riedmiller, and K.
Obermayer, "Autonomous learning of state representations for control:
An emerging field aims to autonomously learn state representations
for reinforcement learning agents from their real-world sensor observations," KI-Künstliche Intelligenz, vol. 29, no. 4, pp. 353-362, 2015. doi:
10.1007/s13218-015-0356-1.
[10] T. Lesort, N. Díaz-Rodríguez, J.-F. Goudou, and D. Filliat,
"State representation learning for control: An overview," Neural

54

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020

Netw., vol.  108, pp. 379-392, Dec. 2018. doi: 10.1016/j.neunet.2018.
07.006.
[11] R. Pérez-Dattari, C. Celemin, J. Ruiz-del Solar, and J. Kober, "Continuous control for high-dimensional state spaces: An interactive
learning approach," in Proc. IEEE Int. Conf. Robotics and Automation
(ICRA), 2019, pp. 7611-7617. doi: 10.1109/ICRA.2019.8793675.
[12] S. Ross, G. Gordon, and D. Bagnell, "A reduction of imitation
learning and structured prediction to no-regret online learning," in
Proc. 14th Int. Conf. Artificial Intelligence and Statistics, 2011, pp.
627-635.
[13] M. Hausknecht and P. Stone, "Deep recurrent Q-learning for partially observable MDPs," in Proc. AAAI Fall Symp. Sequential Decision
Making for Intelligent Agents (AAAI-SDMIA15) Arlington, VA, Nov.
2015.
[14] G. Lample and D. S. Chaplot, "Playing FPS games with deep reinforcement learning," in Proc. 31st AAAI Conf. Artificial Intelligence,
2017, pp. 2140-2146.
[15] D. Ha and J. Schmidhuber, "Recurrent world models facilitate policy evolution," in Proc. Advances Neural Information Processing Systems,
2018, pp. 2450-2462.
[16] A. Zhang, H. Satija, and J. Pineau, Decoupling dynamics and
reward for transfer learning. 2018. [Online]. Available: arXiv:1804.
10689
[17] G. Brockman et al., OpenAI gym. 2016. [Online]. Available:
arXiv:1606.01540
[18] I. Mason, S. Starke, H. Zhang, H. Bilen, and T. Komura, "Fewshot learning of homogeneous human locomotion styles," Comput. Graph. Forum, vol. 37, no. 7, pp. 143-153, 2018. doi: 10.1111/
cgf.13555.
[19] H. Zhang, S. Starke, T. Komura, and J. Saito, "Mode-adaptive neural
networks for quadruped motion control," ACM Trans. Graph. (TOG),
vol. 37, no. 4, pp. 1-11, 2018. doi: 10.1145/3197517.3201366.

Rodrigo Pérez-Dattari, Department of Cognitive Robotics,
Delft University of Technology, The Netherlands. Email:
r.j.perezdattari@tudelft.nl.
Carlos Celemin, Department of Cognitive Robotics, Delft University of Technology, The Netherlands. Email: c.e.celeminpaez
@tudelft.nl.
Giovanni Franzese, Department of Cognitive Robotics, Delft
University of Technology, The Netherlands. Email: g.franzese@
tudelft.nl.
Javier Ruiz-del-Solar, Department of Electrical Engineering
and the Advanced Mining Technology Center, Universidad
de Chile, Santiago. Email: jruizd@ing.uchile.cl.
Jens Kober, Department of Cognitive Robotics, Delft University of Technology, The Netherlands. Email: j.kober@tudelft.nl.
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
