IEEE Robotics & Automation Magazine - September 2023 - 70

This article is concerned with the specific
case of training an object-recognition
system using only a priori visual data to
recognize the same object from the tactile
modality, albeit not previously sensed
through the latter. Humans rely on this
visuo-tactile cross modality and therefore
are able to reconstruct a vision-based
representation of objects and recognize
them using sense of touch only. Similarly,
exploiting this type of cross modality in
robotics would allow learning an object
representation in a controlled environment
using vision and deploying the
system in a more challenging scenario
where the vision is not available (e.g.,
manipulation tasks in clutter, or where
the target objects are not directly visible).
The works proposed in [5] refer to this as cross-modal recognition
(CMR) or visuo-tactile recognition (VTR), taking
inspiration from its psychological definitions [6]. Beyond [5],
other examples of a visuo-tactile cross modality can be found
in the literature [7], [8]. The work presented in [7] explored a
visuo-tactile cross modality to generate tactile images from
visual images and vice versa; yet the training of the two systems
required both tactile and visual data. The work in [8]
used vision to estimate the pose of an object and proposed a
Bayesian algorithm with linear Kalman filters to hone that
prediction with each sequential touch.
To the best of our knowledge, only Falco et al. [5] have per "
POINTNET,
A POINTBASED
ARCHITECTURE,
WAS CHOSEN OVER
PRO JECTION- OR
VOLUMETRIC-BASED
METHODS SUCH
AS A MULTIVIEW
CONVOLUTIONAL NNs
AND VOXNET.
„
descriptors adapt based on the training
dataset and formulation of the learning
task, learning geometric relationships
directly from the data. Rather than proposing
a handcrafted descriptor that can
capture local shapes, we utilize the established
point-based architecture PointNet
[10] to extract task-driven shape descriptors.
The work presented in [11] noted
the gap in the research of partial point
cloud recognition and explored the ability
of PointNet to recognize partial and
noisy point clouds. The authors found
that it was vital to expose the network to
partial representations during training. In
this article, we take it a step further by
formulating a learning task with a training
procedure based on CL [12] to foster
learning of local descriptors from sparser and sparse tactile
data, represented as point clouds.
PointNet, a point-based architecture, was chosen over proformed
CMR by training a system with only the visual modality
on a set of quasi-planar rigid objects. They found point
clouds to be a suitable representation to encode visual and
tactile data for this task. The approach enriches the ensemble
of shape functions (ESF) with information from signature of
histograms of orientations (SHOT) to form the cross-modal
point cLoUd dEscriptor (CLUE) and subsequently use a geodesic
flow kernel (GFK) transfer learning technique to increase
cross-modal performance. The limitation of this work is that
the proposed training pipeline, based on an ensemble of global
handcrafted descriptors for point clouds, requires full tactile
exploration of the object to perform the predictions. However,
as tactile exploration is a time-expensive task, this article
focuses on recognizing objects from partial observations and
making predictions that can be iteratively improved as more
data are gathered. The descriptors employed in [5] are global,
requiring the full tactile model of the object, and therefore are
not directly employable when attempting recognition from partial
observations. Conversely, in this article, we investigate the
use of data-driven techniques to learn more task-specific representations.
Therefore, instead of exploiting existing descriptors,
we define the task and allow the proposed learning procedure
to compose the features.
Neural networks (NNs) have been used extensively for
point cloud recognition in past years to statistically learn
point cloud descriptors or shape embeddings [9]. These
70 IEEE ROBOTICS & AUTOMATION MAGAZINE SEPTEMBER 2023
jection- or volumetric-based methods such as a multiview
convolutional NNs (MVCNNs) and VoxNet [13]. Projectionbased
methods rely on a meshing preprocessing that, besides
being computationally expensive, assumes the emptiness of
the unexplored regions; this is undesirable for recognizing
partially explored shapes. Volumetric-based methods, which
instead construct data structures to represent the occupancy
of a 3D grid and enable the use of 3D convolutions, have
been surpassed by point-based networks [9]. Over the last
few years, PointNet has been influential in deep-shape recognition
as several point-based networks incorporate it for
shape-feature extraction [14] as well as for creating encodings
for generative adversarial networks [15]. Although a
dynamic graph CNN [13], an extension of PointNet, would
also be a good choice, our choice fell on the latter as it is
more computationally efficient because it does not require
the computation of graph structures in latent space. In summary,
current works mainly use tactile sensing alone and
require slow-to-collect tactile datasets, without exploiting
the readiness of visual data. Existing works tackle the problem
by defining handcrafted descriptors. Paganoni et al. [11]
instead studied partial point cloud recognition but derived
samples from the ModelNet40 dataset, which is a highquality
and low-noise CAD dataset. They also analyzed the
performance of PointNet under noise without attempting to
improve it. However, as explained in the " Problem Definition "
section, noise and uncertainties, due to representation
differences, are core to the issue of VTR. Furthermore, in
contrast to the partial point clouds that would be generated
during a tactile exploration, the work in [11] used simulated
laser scans and photogrammetry from single viewpoints. On
the contrary, the tactile point clouds may be composed of
sparse and unconnected clusters of points collected from any
given surface of the object. This study explores decomposing
whole point clouds into patchy partial samples to more
IEEE Robotics & Automation Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2023
