IEEE Robotics & Automation Magazine - September 2023 - 78

the values we used in this article (see the " Training and Evaluation "
section). As is visible from the results listed in Table 3,
ESF and CLUE did not perform well in this scenario. We
argue that there are two reasons for this. First, our dataset contains
only one training sample for each object. Indeed, with
respect to [5], we trained the network with a one-shot learning
approach, using one visual sample for training and one for validation.
The method in [5] does not consider this scenario,
therefore, additional data could be required to train the model.
Second, the authors in [5] consider only planar objects. As previously
explained in the nonplanar case (as in general, the tactile
point cloud cannot be fully explored), the system has to
make a prediction on partial data even when the full tactile
point cloud is considered. As discussed in the " CL " section,
descriptor-based approaches are not suitable for partial recognition,
therefore, they cannot be directly used for cross-modal
recognition of nonplanar objects.
CONCLUSION
This study tackled VTR of the partial point clouds enabled
by a CL procedure applied to only visual data and tested the
network performance on our tactile dataset. We synthesized
partial point clouds and proposed a curriculum that progressively
introduced sparser samples to increase training difficulty
at later epochs. We composed several partial tactile
representations of the objects using our dataset and used
these samples to benchmark the system.
The curriculum-trained network was able to perform recognition
with an accuracy of 80% using only 20% of points
generated from a random exploration strategy. The accuracy
further increased to 100% on clouds using at least 60% of the
data. In contrast, the vanilla-trained network averaged an accuracy
of 90.7% for ..p06
$$1 We benchmarked the inference
time of the pipeline on our hardware, which refreshed predictions
at a promising rate of 47 Hz.
We also treated the tactile data collected, as described in
the " Experiments " section, as a time-ordered sequence of
points and showed that CL improved the system's accuracy for
online recognition, during exploration, by an average of 11%.
Our experiments also highlighted the impact of the exploration
strategy on overall system performance. A natural extension
would study exploration strategies to minimize the number of
contacts required to recognize an object with high confidence.
ACKNOWLEDGMENT
We gratefully acknowledge support from the Engineering
and Physical Sciences Research Council program grant From
Sensing to Collaboration (EP/V000748/1). This article has
supplementary information downloadable at https://doi.
org/10.1109MRA.2022.1003122204. Alessandro Albini is the
corresponding author.
AUTHORS
Christopher Parsons, Oxford Robotics Institute, University
of Oxford, Oxford OX2 6NN, United Kingdom. E-mail:
christopher.parsons@bnc.ox.ac.uk.
78 IEEE ROBOTICS & AUTOMATION MAGAZINE SEPTEMBER 2023
Alessandro Albini, Oxford Robotics Institute, University
of Oxford, Oxford OX2 6NN, United Kingdom. E-mail:
alessandro@robots.ox.ac.uk.
Daniele De Martini, Oxford Robotics Institute,
University of Oxford, Oxford OX2 6NN, United Kingdom.
E-mail: daniele.demartini@eng.ox.ac.uk.
Perla Maiolino, Oxford Robotics Institute, University of
Oxford, Oxford OX2 6NN, United Kingdom. E-mail: perla.
maiolino@eng.ox.ac.uk.
REFERENCES
[1] H. Liu, Y. Wu, F. Sun, and D. Guo, " Recent progress on tactile object recognition, "
Int. J. Adv. Robot. Syst., vol. 14, no. 4, p. 1,729,881,417,717,056, 2017, doi:
10.1177/1729881417717056.
[2] H. Liu, F. Sun, B. Fang, and D. Guo, " Cross-modal zero-shot-learning for tactile
object recognition, " IEEE Trans. Syst., Man, Cybern. Syst., vol. 50, no. 7, pp.
2466-2474, Jul. 2020, doi: 10.1109/TSMC.2018.2818184.
[3] S. Wang et al., " 3D shape perception from monocular vision, touch, and shape
priors, " in Proc. IEEE Int. Conf. Intell. Robots Syst., 2018, pp. 1606-1613, doi:
10.1109/IROS.2018.8593430.
[4] L. Rustler, J. Lundell, J. K. Behrens, V. Kyrki, and M. Hoffmann, " Active
visuo-haptic object shape completion, " IEEE Robot. Autom. Lett., vol. 7, no. 2, pp.
5254-5261, Apr. 2022, doi: 10.1109/LRA.2022.3152975.
[5] P. Falco, S. Lu, C. Natale, S. Pirozzi, and D. Lee, " A transfer learning approach
to cross-modal object recognition: From visual observation to robotic haptic
exploration, " IEEE Trans. Robot., vol. 35, no. 4, pp. 987-998, Aug. 2019, doi:
10.1109/TRO.2019.2914772.
[6] R. Blake, K. V. Sobel, and T. W. James, " Neural synergy between kinetic
vision and touch, " Psychological Sci., vol. 15, no. 6, pp. 397-402, Jun. 2004, doi:
10.1111/j.0956-7976.2004.00691.x.
[7] J. T. Lee, D. Bollegala, and S. Luo, " 'Touching to see' and 'seeing to feel':
Robotic cross-modal sensory data generation for visual-tactile perception, " in
Proc. IEEE Int. Conf. Robot. Autom., 2019, vol. 2019, pp. 4276-4282, doi:
10.1109/ICRA.2019.8793763.
[8] P. K. Murali, M. Gentner, and M. Kaboli, " Active visuo-tactile point cloud registration
for accurate pose estimation of objects in an unknown workspace, " 2021.
[Online]. Available: http://arxiv.org/abs/2108.04015
[9] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, " Deep learning
for 3D point clouds: A survey, " 2019, arXiv:1912.12033.
[10] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, " PointNet: Deep learning on point
sets for 3D classification and segmentation, " in Proc. 30th IEEE Conf. Comput.
Vis. Pattern Recognit., 2017, vol. 2017-Janua, pp. 652-660.
[11] S. Paganoni, E. Zappa, and S. Turrisi, " Classification reliability of 3D shapes using
neural networks in case of partial and noisy models, " in Proc. I2MTC 2020 - Int.
Instrum. Meas. Technol. Conf., pp. 1-6, doi: 10.1109/I2MTC43012.2020.9128469.
[12] A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu,
" Automated curriculum learning for neural networks, " in Proc. 34th Int. Conf.
Mach. Learn., 2017, vol. 3, pp. 2120-2129.
[13] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, " Deep learning
for 3D point clouds: A survey, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 43,
no. 12, pp. 4338-4364, Dec. 2021, doi: 10.1109/TPAMI.2020.3005434.
[14] Y. Duan, Y. Zheng, J. Lu, J. Zhou, and Q. Tian, " Structural relational reasoning
of point clouds, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern
Recognit., 2019, vol. 2019-June, pp. 949-958, doi: 10.1109/CVPR.2019.00104.
[15] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas, " Learning representations
and generative models for 3D point clouds, " in Proc. 35th Int. Conf. Mach.
Learn., 2018, vol. 1, pp. 67-85.
[16] R. S. Dahiya, G. Metta, M. Valle, and G. Sandini, " Tactile sensing-from
humans to humanoids, " IEEE Trans. Robot., vol. 26, no. 1, pp. 1-20, Feb. 2010,
doi: 10.1109/TRO.2009.2033627.
[17] M. Alexa, J. Behr, D. Cohen-Or, S. Fleishman, D. Levin, and C. T. Silva,
" Computing and rendering point set surfaces, " IEEE Trans. Vis. Comput. Graphics,
vol. 9, no. 1, pp. 3-15, Jan./Mar. 2003, doi: 10.1109/TVCG.2003.1175093.
[18] S. Salti, F. Tombari, and L. D. Stefano, " SHOT: Unique signatures of histograms
for surface and texture description, " Comput. Vis. Image Understanding, vol.
125, pp. 251-264, Aug. 2014, doi: 10.1016/j.cviu.2014.04.011.
[19] W. Wohlkinger and M. Vincze, " Ensemble of shape functions for 3D object
classification, " in Proc. 2011 IEEE Int. Conf. Robot. Biomimetics, pp. 2987-2992,
doi: 10.1109/ROBIO.2011.6181760.
[20] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, " Generalizing from a few examples:
A survey on few-shot learning, " ACM Comput. Surv., vol. 53, no. 3, pp. 1-34,
2020, doi: 10.1145/3386252.
http://arxiv.org/abs/2108.04015 https://doi.org/10.1109MRA.2022.1003122204 https://doi.org/10.1109MRA.2022.1003122204
IEEE Robotics & Automation Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2023
