IEEE Robotics & Automation Magazine - June 2020 - 128

algorithm [2]. The computational limitations of sparse GPs can
be mitigated, to an extent, by the use of sparse GP approximations. We also provided a head-to-head comparison between
the two algorithms presented here. GP-VI-MFRL (the modelbased version) performs better than GPQ-MFRL (the modelfree version) in the beginning, which is consistent with the
outcomes for traditional RL techniques, where model-based
algorithms tend to perform better than model-free ones.
An immediate future work is to compare the MFRL
technique with sim2real approaches [1]. Unlike sim2real,
the presented MFRL techniques explicitly decide when to
switch between simulators and use more than two levels
of simulators. An interesting avenue of future work
would be to combine the two ideas: use MFRL to model
the fact that some simulators are cheaper/faster to operate than others and use parameterized simulators to
bring in domain adaptation/randomization for better
generalization. Finally, in the current approach, data
from different simulators are not combined when GP
regression is performed.
One possibility for improvement is to use multitask GPs
that can simultaneously produce multiple outputs, one corresponding to each fidelity simulator. Multitask GPs [21] can
produce multiple outputs simultaneously, one corresponding
to each simulator. An alternative is to use deep GPs to combine data from various fidelities as part of the same network.
In both cases, the goal is to learn the correlation between values in different environments directly.
Acknowledgments
This work has been funded by the Center for Unmanned
Aircraft Systems (C-UAS), a National Science Foundation-
sponsored industry/university cooperative research center
under NSF award IIP-1161036, along with significant
contributions from C-UAS industry members. This work
was performed when the authors were with the Department of Electrical and Computer Engineering, Virginia
Tech, Blacksburg.

vol. 6, pp. 1939-1959, Dec. 2005. doi: 10.5555/1046920.1194909.
[7] J. Kober, J. A. Bagnell, and J. Peters, "Reinforcement learning in
robotics: A survey," Int. J. Robot. Res., vol. 32, no. 11, pp. 1238-1274,
2013. doi: 10.1177/0278364913495721.
[8] J. Tan et al., Sim-to-real: Learning agile locomotion for quadruped
robots. 2018. [Online]. Available: https://arxiv.org/pdf/1804.10332.pdf
[9] M. C. Kennedy and A. O'Hagan, "Predicting the output from a complex computer code when fast approximations are available," Biometrika, vol. 87, no. 1, pp. 1-13, 2000. doi: 10.1093/biomet/87.1.1.
[10] P. Perdikaris, M. Raissi, A. Damianou, N. Lawrence, and G. E. Karniadakis, "Nonlinear information fusion algorithms for data-efficient
multi-fidelity modelling," Proc. R. Soc. A, Math. Phys. Eng. Sci., vol.
473, no. 2198, p. 20160751, 2017. doi: 10.1098/rspa.2016.0751.
[11] A. Damianou, "Deep Gaussian processes and variational propagation
of uncertainty," Ph.D. dissertation, Dept. Neurosci., Univ. Sheffield, 2015.
[12] A. Marco et al., "Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with Bayesian optimization," in Proc. 2017 IEEE Int. Conf. Robotics and Automation (ICRA).
pp. 1557-1563. doi: 10.1109/ICRA.2017.7989186.
[13] P. Hennig and C. J. Schuler, "Entropy search for information-efficient
global optimization," J. Mach. Learn. Res., vol. 13, pp. 1809-1837, June 2012.
[14] J. Tobin et al., "Domain randomization for transferring deep neural networks from simulation to the real world," in Proc. 2017 IEEE/RSJ
Int. Conf. Intelligent Robots and Systems (IROS), 2017, pp. 23-30. doi:
10.1109/IROS.2017.8202133.
[15] T. Jung and P. Stone, "Gaussian processes for sample efficient reinforcement learning with RMAX-like exploration," in Proc. European
Conf. Machine Learning, Sept. 2010, pp. 601-616. [Online]. Available:
http://www.cs.utexas.edu/users/ai-lab/?ECML10-jung
[16] R. I. Brafman and M. Tennenholtz, "R-max-a general polynomial
time algorithm for near-optimal reinforcement learning," J. Mach. Learn.
Res., vol. 3, pp. 213-231, Oct. 2002. doi: 10.1162/153244303765208377.
[17] R. Grande, T. Walsh, and J. How, "Sample efficient reinforcement
learning with Gaussian processes," in Proc. 31st Int. Conf. Machine
Learning (ICML-14), 2014, pp. 1332-1340. doi: 10.5555/3044805.3045041.
[18] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduc[19] C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for

[1] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, and N.

Machine Learning. Cambridge, MA: MIT Press, 2006.

Ratliff, "Closing the sim-to-real loop: Adapting simulation randomiza-

[20] E. Bingham et al., Pyro: Deep universal probabilistic program-

tion with real world experience," in Proc. Int. Conf. Robotics and Auto-

ming. 2018. [Online]. Available: https://arxiv.org/pdf/1810.09538.pdf

mation (ICRA), May 2019. doi: 10.1109/ICRA.2019.8793789.

[21] E. V. Bonilla, K. M. Chai, and C. Williams, "Multi-task Gaussian

[2] M. Cutler, T. J. Walsh, and J. P. How, "Real-world reinforcement

process prediction," in Proc. Advances Neural Information Processing

learning via multifidelity simulators," IEEE Trans. Robot., vol. 31, no. 3,

Systems, 2008, pp. 153-160. doi: 10.5555/2981562.2981582.

pp. 655-671, June 2015. doi: 10.1109/TRO.2015.2419431.
[3] M. E. Taylor, P. Stone, and Y. Liu, "Transfer learning via inter-task
mappings for temporal difference learning," J. Mach. Learn. Res., vol. 8,
pp. 2125-2167, Sept. 2007.
[4] M. Cutler and J. P. How, "Efficient reinforcement learning for robots
using informative simulated priors," in Proc. 2015 IEEE Int. Conf.
Robotics and Automation (ICRA), 2015, pp. 2605-2612. doi: 10.1109/
ICRA.2015.7139550.
[5] R. M. Neal, Bayesian Learning for Neural Networks, vol. 118, Berlin:
Springer-Verlag, 2012.

*

sparse approximate Gaussian process regression," J. Mach. Learn. Res.,

tion. Cambridge, MA: MIT Press, 1998.

References

128

[6] J. Quiñonero-Candela and C. E. Rasmussen, "A unifying view of

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020

Varun Suryan, Department of Computer Science, University
of Maryland College Park. Email: suryan@umd.edu.
Nahush Gondhalekar, Qualcomm, Inc., San Diego, California.
Email: nahushg@vt.edu.
Pratap Tokekar, Department of Computer Science, University
of Maryland College Park. Email: tokekar@umd.edu.
https://www.arxiv.org/pdf/1804.10332.pdf http://www.cs.utexas.edu/users/ai-lab/?ECML10-jung https://www.arxiv.org/pdf/1810.09538.pdf
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
