IEEE Robotics & Automation Magazine - September 2023 - 69

Partial Point Clouds
Visuo-Tactile
Recognition of
Using PointNet
and Curriculum
Learning
Enabling Tactile Perception from Visual Data
By Christopher Parsons , Alessandro Albini ,
Daniele De Martini , and Perla Maiolino
This article is about recognizing handheld objects from
incomplete tactile observations with a classifier trained on
only visual representations. Our method is based on the deep
learning (DL) architecture PointNet and a curriculum learning
(CL) technique for fostering the learning of descriptors robust
to partial representations of objects. The learning procedure
gradually decomposes the visual point clouds to synthesize
sparser and sparser input data for the model. In this manner,
we were able to employ one-shot learning, using the decomposed
visual point clouds as augmentations, and reduce the
data-collection requi rement for training. The approach allows
for a gradual improvement of prediction accuracy as more tactile
data become available.
We evaluated the effectiveness of the curriculum strategy on
our generated visual and tactile datasets, experimentally showing
that the proposed method improved recognition accuracy
by up to 23% on partial tactile data and boosted accuracy on
full tactile data from 93 to 100%. The curriculum-trained network
recognized objects with an accuracy of 80% using only
20% of the tactile data representing the objects,
Digital Object Identifier 10.1109/MRA.2022.3212316
Date of current version: 21 October 2022
1070-9932/22©2022IEEE
increasing to 100% accuracy on clouds containing at least 60%
of the points.
INTRODUCTION
Sense of touch is crucial for humans in performing several
tasks, from exploration to manipulation and object recognition.
In literature, computer vision remains the dominant
modality for object-recognition tasks; nevertheless, touch can
provide crucial information in unstructured environments
when vision is impaired due to poor lighting, unfavorable
weather, translucent objects, or occlusion. This has motivated
researchers in exploring not only tactile-based object recognition
[1] but also combining vision and tactile information to
support vision-based perception [2] and accurately reconstructing
the shape of complex 3D objects, even when vision is
subjected to occlusions [3], [4]. However, these methods
require tactile-based models of the objects for training, which
can only be obtained through time and resource expensive
exploration tasks. In this respect, it would be convenient to 1)
exploit the readiness and availability of visual data to train a
system that can recognize objects through environment-robust
touch and 2) recognize an object using partial data to mitigate
the need to complete a long and expensive tactile exploration.
SEPTEMBER 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
69
ALL IMAGES LICENSED BY INGRAM PUBLISHING

https://orcid.org/0000-0002-6828-3065 https://orcid.org/0000-0003-1562-7044 https://orcid.org/0000-0001-6121-5839 https://orcid.org/0000-0002-7588-9567

IEEE Robotics & Automation Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2023