IEEE Robotics & Automation Magazine - June 2020 - 138

Our current pose-prediction performance depends on the
accuracy of the frame prediction in RGB image space, which
may become more challenging as weather and lighting conditions change. Future work will include investigating human pose
prediction based on alternative or complementary sensors, such
as depth and thermal imaging cameras. With the development
of advanced computing units, network architecture design and
automated parameter settings for improving the efficiency and
scalability of DL approaches can be further studied as well.
Finally, it is essential to develop evaluation metrics for pose
prediction, particularly when GT is not available or imprecise.
An open problem remains as to how to evaluate whether the
predicted pedestrian poses are natural and realistic and
whether they are in accordance with traffic rules and regulations as well as ethical considerations.
Acknowledgments
This work was supported by a grant from the Ford Motor
Company via the Ford-University of Michigan Alliance
under award N022884.
References
[1] W. Burgard et al., "The interactive museum tour-guide robot," in Proc.
15th Nat./10th Conf. Artificial Intelligence/Innovative Applications of Artificial Intelligence, 1998, pp. 11-18. doi: 10.5555/295240.295249.
[2] S. Thrun et al., "MINERVA: A second-generation museum tourguide robot," in Proc. IEEE Int. Conf. Robotics and Automation (ICRA),
1999, vol. 3, pp. 1999-2005. doi: 10.1109/ROBOT.1999.770401.
[3] V. Villani, F. Pini, F. Leali, and C. Secchi, "Survey on human-robot
collaboration in industrial settings: Safety, intuitive interfaces and
applications," Mechatronics, vol. 55, pp. 248-266, Nov. 2018. doi:
10.1016/j.mechatronics.2018.02.009.
[4] W. Kim et al., "Adaptable workstations for human-robot collaboration: A reconfigurable framework for improving worker ergonomics
and productivity," IEEE Robot. Autom. Mag, vol. 26, no. 3, pp. 14-26,
2019. doi: 10.1109/MRA.2018.2890460.
[5] I. Petrov, V. Shakhuro, and A. Konushin, "Deep probabilistic human
pose estimation," IET Comput. Vis., vol. 12, no. 5, pp. 578-585, 2018. doi:
10.1049/iet-cvi.2017.0382.
[6] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik, "Recurrent network models for human dynamics," in Proc. IEEE Int. Conf. Computer
Vision (ICCV), 2015, pp. 4346-4354. doi: 10.1109/ICCV.2015.494.
[7] J. Bütepage, H. Kjellström, and D. Kragic, "Anticipating many
futures: Online human motion prediction and generation for humanrobot interaction," in Proc. IEEE Int. Conf. Robotics and Automation
(ICRA), 2018, pp. 1-9. doi: 10.1109/ICRA.2018.8460651.
[8] S. Hochreiter and J. Schmidhuber, "Long short-term memory,"
Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997. doi: 10.1162/
neco.1997.9.8.1735.
[9] W. Lotter, G. Kreiman, and D. Cox, "Deep predictive coding networks
for video prediction and unsupervised learning," in Proc. Int. Conf. Learning Representations, 2017. [Online]. Available: https://openreview.net/
pdf?id=B1ewdt9xe
[10] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, "Vision meets robotics: The KITTI dataset," Int. J. Robot. Res., vol. 32, no. 11, pp. 1231-1237,
2013. doi: 10.1177/0278364913491297.

138

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020

[11] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh,
"OpenPose: Realtime multi-person 2D pose estimation using part
affinity fields," IEEE Trans. Pattern Anal. Mach. Intell., to be published. doi: 10.1109/TPAMI.2019.2929257.
[12] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask R-CNN," in
Proc. IEEE Int. Conf. Computer Vision (ICCV), 2017, pp. 2980-2988.
doi: 10.1109/ICCV.2017.322.
[13] I. Kotseruba, A. Rasouli, and J. K. Tsotsos, Joint attention in autonomous
driving (JAAD). 2016. [Online]. Available: https://arxiv.org/abs/1609.04741
[14] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, "Agreeing to cross: How
drivers and pedestrians communicate," in Proc. IEEE Intell. Veh. Symp.
(IV), 2017, pp. 264-269. doi: 10.1109/IVS.2017.7995730.
[15] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, "Are they going to cross?
A benchmark dataset and baseline for pedestrian crosswalk behavior," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2017, pp. 206-213.
doi: 10.1109/ICCVW.2017.33.
[16] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, "Understanding pedestrian behavior in complex traffic scenes," IEEE Trans. Intell. Veh., vol. 3,
no. 1, pp. 61-70, 2017. doi: 10.1109/TIV.2017.2788193.
[17] W. Kim et al., "PedX: Benchmark dataset for metric 3-D pose estimation of pedestrians in complex urban intersections," IEEE Robot.
Autom. Lett., vol. 4, no. 2, pp. 1940-1947, Apr. 2019. doi: 10.1109/
LRA.2019.2896705. [Online]. Available: http://pedx.io/
[18] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image
quality assessment: From error visibility to structural similarity," IEEE
Trans. Image Process., vol. 13, no. 4, pp. 600-612, 2004. doi: 10.1109/
TIP.2003.819861.
[19] X. Du, R. Vasudevan, and M. Johnson-Roberson, "Bio-LSTM: A
biomechanically inspired recurrent neural network for 3-D pedestrian
pose and gait prediction," IEEE Robot. Autom. Lett., vol. 4, no. 2, pp.
1501-1508, Apr. 2019. doi: 10.1109/LRA.2019.2895266.
[20] F. Chollet et al., "Keras: The Python deep learning library," Keras,
2015. [Online]. Available: https://keras.io
[21] F. Bonsignorio, E. Messina, A. P. del Pobil, and J. Hallam, Eds., "The
road ahead-final remarks," in Metrics of Sensory Motor Coordination and
Integration in Robots and Animals: How to Measure the Success of Bioinspired Solutions with Respect to Their Natural Models, and Against More
'Artificial' Solutions?, vol. 36, 2020, pp. 181-185. [Online]. Available: https://
link.springer.com/content/pdf/10.1007/978-3-030-14126-4.pdf#page=201
[22] CMU Perceptual Computing Lab/OpenPose. Accessed on: Feb. 27, 2020.
[Online]. Available: https://github.com/CMU-Perceptual-Computing-Lab/
openpose/blob/master/doc/quick_start.md#quick-start
[23] Mask_RCNN_Humanpose. Accessed on: Feb. 27, 2020. [Online]. Available: https://github.com/Superlee506/Mask_RCNN_Humanpose/releases

Xiaoxiao Du, Department of Naval Architecture and Marine
Engineering, University of Michigan, Ann Arbor. Email:
xiaodu@umich.edu.
Ram Vasudevan, Department of Mechanical Engineering,
University of Michigan, Ann Arbor. Email: ramv@umich.edu.
Matthew Johnson-Roberson, Department of Naval Architecture
and Marine Engineering, University of Michigan, Ann Arbor.
Email: mattjr@umich.edu.

https://www.arxiv.org/abs/1609.04741 http://pedx.io/ https://keras.io/ https://link.springer.com/content/pdf/10.1007/978-3-030-14126-4.pdf#page=201 https://link.springer.com/content/pdf/10.1007/978-3-030-14126-4.pdf#page=201 https://www.github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/quick_start.md#quick-start https://www.github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/quick_start.md#quick-start https://www.github.com/Superlee506/Mask_RCNN_Humanpose/releases https://www.openreview.net/pdf?id=B1ewdt9xe https://www.openreview.net/pdf?id=B1ewdt9xe

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020