IEEE Robotics & Automation Magazine - March 2022 - 57

several pieces of open source software (see the " Sharing Spatial
Information " section) to help facilitate research in this
domain that interfaces HoloLens 2 devices and robots.
Within immersive teleoperation, some of the major open
questions will need to be resolved through better human
understanding. The user studies around motion retargeting
and task autocorrection have, so far, been limited in scope.
Consequently, there are questions about whether it is possible
and beneficial to provide personalized models for every user
to provide an improved experience from a single method.
Deeper and broader user studies should be performed to better
understand the requirements for general usability and provide
insights on future research questions to pursue.
Bringing spatial and human understanding together in a
human-robot system offers promising new modes of interaction.
We have shown several MR-based systems that leverage
these technologies to provide solutions to real-world robot
use cases. While these capabilities are deployable today and
applicable to many commercial and research scenarios
beyond the scope of this article, these works represent the tip
of the iceberg, with many possibilities for future research in
spatial computing and human-robot interaction.
Acknowledgments
The authors would like to thank Oswaldo Ferro, Jan Stühmer,
Lukas Gruber, and Simon Huber for their contributions to
these projects.
References
[1] S. Greenwold, Spatial Computing. Cambridge, MA, USA: MIT
Press, 2003.
[2] M. Gianni, G. Gonnelli, A. Sinha, M. Menna, and F. Pirri, " An augmented
reality approach for trajectory planning and control of tracked
vehicles in rescue environments, " in Proc. 2013 IEEE Int. Symp. Saf.,
Security, Rescue Robot. (SSRR), pp. 1-6, doi: 10.1109/SSRR.2013.6719360.
[3] D. Vaquero Melchor, " Technologies and concepts for enhancing interaction
and enabling collaboration in augmented and mixed reality, " Ph.D.
dissertation, Universidad Politécnica de Madrid, Madrid, Spain, 2020.
[4] Y. Liu et al., " Usability evaluation for drone mission planning in
virtual reality, " in Proc. Int. Conf. Virtual, Augmented Mixed Reality,
2018, pp. 313-330, doi: 10.1007/978-3-319-91584-5_25.
[5] H. Fang, S. K. Ong, and A. Y.-C. Nee, " Robot programming using
augmented reality, " in Proc. 2009 IEEE Int. Conf. CyberWorlds, pp.
13-20, doi: 10.1109/CW.2009.14.
[6] M. Ostanin and A. Klimchik, " Interactive robot programing using
mixed reality, " IFAC-PapersOnLine, vol. 51, no. 22, pp. 50-55, 2018, doi:
10.1016/j.ifacol.2018.11.517.
[7] J. Delmerico et al., " The current state and future outlook of rescue
robotics, " J. Field Robot., vol. 36, no. 1, pp. 1171-1191, 2019, doi: 10.1002/
rob.21887.
[8] K. Chandan, V. Kudalkar, X. Li, and S. Zhang, " ARROCH: Augmented
reality for robots collaborating with a human, " in Proc. IEEE Int.
Conf. Robot. Automat. (ICRA), 2021, pp. 3787-3793, doi: 10.1109/
ICRA48506.2021.9561144.
[9] F. Muhammad, A. Hassan, A. Cleaver, and J. Sinapov, " Creating a
shared reality with robots, " in Proc. IEEE ACM/IEEE Int. Conf.
Human-Robot Interact. (HRI), 2019, pp. 614-615, doi: 10.1109/
HRI.2019.8673191.
[10] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge,
MA, USA: MIT Press, 2016.
[11] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical
Learning: Data Mining, Inference, and Prediction. New York:
Springer-Verlag, 2009.
[12] A. Vaswani et al., " Attention is all you need, " in Proc. Adv. Neural
Inf. Process. Syst., 2017, pp. 6000-6010.
[13] A. Handa et al., " DexPilot: Vision-based teleoperation of dexterous
robotic hand-arm system, " in Proc. 2020 IEEE Int. Conf. Robot.
Automat. (ICRA), pp. 9164-9170, doi: 10.1109/ICRA40945.2020.9197124.
[14] D. Rakita, B. Mutlu, and M. Gleicher, " A motion retargeting method
for effective mimicry-based teleoperation of robot arms, " in Proc. 2017
12th ACM/IEEE Int. Conf. Human-Robot Interact., pp. 361-370, doi:
10.1145/2909824.3020254.
[15] S. Zimmermann, G. Hakimifard, M. Zamora, R. Poranne, and S.
Coros, " A multi-level optimization framework for simultaneous grasping
and motion planning, " IEEE Robot. Autom. Lett., vol. 5, no. 2, pp.
2966-2972, 2020, doi: 10.1109/LRA.2020.2974684.
[16] A. Dragan and S. Srinivasa, " Formalizing assistive teleoperation, " in
Proc. Robot., Sci. Syst., pp. 73-80, Jul. 2012, doi: 10.15607/RSS.2012.VIII.010.
[17] D. Kent, C. Saldanha, and S. Chernova, " A comparison of remote
robot teleoperation interfaces for general object manipulation, " in Proc.
2017 ACM/IEEE Int. Conf. Human-Robot Interact., pp. 371-379, doi:
10.1145/2909824.3020249.
[18] H. Hedayati, M. Walker, and D. Szafir, " Improving collocated robot
teleoperation with augmented reality, " in Proc. ACM/IEEE Int. Conf.
Human-Robot Interact., 2018, pp. 78-86, doi: 10.1145/3171221.3171251.
Jeffrey Delmerico, Microsoft MR and AI Lab, Zürich, 8001,
Switzerland. Email: jeffrey.delmerico@microsoft.com.
Roi Poranne, Computational Robotics Lab, ETH Zürich,
Zürich, 8092, Switzerland, and Department of Computer Science,
University of Haifa, Haifa, 3498838, Israel. Email: roi
.poranne@inf.ethz.ch.
Federica Bogo, Microsoft MR and AI Lab, Zürich, 8001, Switzerland.
Email: federica.bogo@microsoft.com.
Helen Oleynikova, NVIDIA, Zürich, 8004, Switzerland. Email:
holeynikova@nvidia.com.
Eric Vollenweider, Microsoft MR and AI Lab, Zürich, 8001,
Switzerland. Email: eric.vollenweider@microsoft.com.
Stelian Coros, Computational Robotics Lab, ETH Zürich,
Zürich, 8092, Switzerland. Email: scoros@ inf.ethz.ch.
Juan Nieto, Microsoft MR and AI Lab, Zürich, 8001, Switzerland.
Email: juan.nieto@microsoft.com.
Marc Pollefeys, Microsoft MR and AI Lab, Zürich, 8001, Switzerland.
Email: marc.pollefeys@microsoft.com.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
57
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
