IEEE Robotics & Automation Magazine - June 2020 - 77

perceptual system with invariance to contact motion. As
demonstrated here, appropriately trained deep NNs can solve
that problem.
Acknowledgments
We thank NVIDIA for the donation of the Titan Xp GPU
used for this research. This work was supported by an award
from the Leverhulme Trust for "A Biomimetic Forebrain for
Robot Touch" (RL-2016-39). Nathan F. Lepora and John
Lloyd contributed equally to this work.
References
[1] N. Lepora, A. Church, C. de Kerckhove, R. Hadsell, and J. Lloyd,
"From pixels to percepts: Highly robust edge perception and contour
following using deep learning and an optical biomimetic tactile sensor," IEEE Robot. Autom. Lett., vol. 4, no. 2, pp. 2101-2107, 2019. doi:
10.1109/LRA.2019.2899192.
[2] W. Yuan, S. Wang, S. Dong, and E. Adelson, "Connecting look and
feel: Associating the visual and tactile properties of physical materials," in Proc. IEEE Conf. Computer Vision and Pattern Recognition
(CVPR), 2017, pp. 4494-4502. doi: 10.1109/CVPR.2017.478.
[3] A. Schmitz, Y. Bansho, K. Noda, H. Iwata, T. Ogata, and S. Sugano,
"Tactile object recognition using deep learning and dropout," in Proc.
IEEE-RAS Int. Conf. Humanoid Robots, 2014, pp. 1044-1050. doi:
10.1109/HUMANOIDS.2014.7041493.
[4] L. Cao et al., "Efficient spatio-temporal tactile object recognition
with randomized tiling convolutional networks in a hierarchical
fusion strategy," in Proc. AAAI Conf. Artificial Intelligence, 2015, pp.
3337-3345.
[5] S. Baishya and B. Bäuml, "Robust material classification with a tactile
skin using deep learning," in Proc. IEEE/RSJ Int. Conf. Intelligent Robots
and Systems (IROS), 2016, pp. 8-15. doi: 10.1109/IROS.2016.7758088.
[6] M. Meier, F. Patzelt, R. Haschke, and H. Ritter, "Tactile convolutional networks for online slip and rotation detection," in Proc. Int. Conf.
Artificial Neural Networks (ICANN), 2016, pp 12-19. doi: 10.1007/978-3319-44781-0_2.
[7] G. Buscher, M. Meier, G. Walck, R. Haschke, and H. Ritter, "Augmenting curved robot surfaces with soft tactile skin," in Proc. 2015
IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pp. 1514-
1519. doi: 10.1109/IROS.2015.7353568.
[8] Y. Gao, L. Hendricks, K. Kuchenbecker, and T. Darrell, "Deep learning for tactile understanding from visual and haptic data," in Proc.
IEEE Int. Conf. Robotics and Automation, 2016, pp. 536-543. doi:
10.1109/ICRA.2016.7487176.
[9] F. Sun, C. Liu, W. Huang, and J. Zhang, "Object classification and
grasp planning using visual and tactile sensing," IEEE Trans. Syst.,
Man, Cybern., Syst., vol. 46, no. 7, pp. 969-979, 2016. doi: 10.1109/
TSMC.2016.2524059.
[10] R. Calandra et al., "More than a feeling: Learning to grasp and
regrasp using vision and touch," IEEE Robot. Autom. Lett., vol. 3, no. 4,
pp. 3300-3307, 2018. doi: 10.1109/LRA.2018.2852779.
[11] S. Luo, W. Yuan, E. Adelson, A. Cohn, and R. Fuentes, "ViTac: Feature sharing between vision and tactile sensing for cloth texture recognition," in Proc. IEEE Int. Conf. Robotics and Automation, pp. 2722-
2727. 2018. doi: 10.1109/ICRA.2018.8460494.

[12] J. Lee, D. Bollegala, and S. Luo, "Touching to see" and "seeing to
feel": Robotic cross-modal sensory data generation for visual-tactile
perception," in Proc. 2019 Int. Conf. Robotics and Automation (ICRA),
2019, pp. 4276-4282. doi: 10.1109/ICRA.2019.8793763.
[13] J. Lin, R. Calandra, and S. Levine, "Learning to identify object
instances by touch: Tactile recognition via multimodal matching," in
Proc. Int. Conf. Robotics and Automation (ICRA), 2019, pp. 3644-3650.
doi: 10.1109/ICRA.2019.879388.
[14] J. James, A. Church, L. Cramphorn, and N. Lepora, Tactile model
O: Fabrication and testing of a 3d-printed, three-fingered tactile robot
hand. 2019. [Online]. Available: https:arXiv:1907.07535
[15] F. Hogan, M. Bauza, O. Canal, E. Donlon, and A. Rodriguez, "Tactile regrasp: Grasp adjustments via simulated tactile transformations,"
in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS),
2018, pp. 2963-2970. doi: 10.1109/IROS.2018.8593528.
[16] P. Dario and D. De Rossi, "Tactile sensors and the gripping challenge," IEEE Spectr., vol. 22, no. 8, pp. 46-53, 1985. doi: 10.1109/
MSPEC.1985.6370785.
[17] W. Yuan, C. Zhu, A. Owens, M. Srinivasan, and E. Adelson. "Shapeindependent hardness estimation using deep learning and a GelSight
tactile sensor," in Proc. IEEE Int. Conf. Robotics and Automation
(ICRA), 2017, pp. 951-958. doi: 10.1109/ICRA.2017.7989116.
[18] S. Tian et al., "Manipulation by feel: Touch-based control with
deep predictive models," in Proc. Int. Conf. Robotics and Automation
(ICRA), 2019, pp. 818-824. doi: 10.1109/ICRA.2019.8794219.
[19] R Calandra et al., "The feeling of success: Does touch sensing help
predict grasp outcomes?" in Proc. Conf. Robot Learning (CoRL), 2017,
pp. 1-10.
[20] M. Polic, I. Krajacic, N. Lepora, and M. Orsag, "Convolutional autoencoder for feature extraction in tactile sensing," IEEE Robot. Autom.
Lett., vol. 4, no. 4, p. 1, 2019. doi: 10.1109/LRA.2019.2927950.
[21] M. Johnson and E. Adelson, "Retrographic sensing for the measurement of surface texture and shape," in Proc. 2009 IEEE Computer
Society Conf. Computer Vision and Pattern Recognition Workshops,
CVPR Workshops, pp. 1070-1077. doi: 10.1109/CVPR.2009.5206534.
[22] C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, "Development of a
tactile sensor based on biologically inspired edge encoding," in Proc.
Int. Conf. Advanced Robotics, 2009, pp. 1-6.
[23] W. Yuan, S. Dong, and E. Adelson, "GelSight: High-resolution robot
tactile sensors for estimating geometry and force," Sensors, vol. 17, no.
12, p. 2762, 2017. doi: 10.3390/s17122762.
[24] B. Ward-Cherrier et al., "The TacTip family: Soft optical tactile
sensors with 3d-printed biomimetic morphologies," Soft Robot., vol. 5,
no. 2, pp. 216-227, 2018. doi: 10.1089/soro.2017.0052.

Nathan F. Lepora, Department of Engineering Mathematics,
Faculty of Engineering, University of Bristol and Bristol
Robotics Laboratory, United Kingdom. Email: n.lepora@bris
tol.ac.uk.
John Lloyd, Department of Engineering Mathematics, Faculty
of Engineering, University of Bristol and Bristol Robotics
Laboratory, United Kingdom. Email: jl15313@bristol.ac.uk.

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

77
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
