IEEE Robotics & Automation Magazine - September 2023 - 72

the possibility of performing touch-based predictions online,
thus avoiding the whole exploration of the object.
METHODOLOGY
To address the problem of recognizing an object from its
partial tactile representation, we form a custom CL procedure
(see the " CL " section), which exposes the network to
synthetic partial point clouds. We hypothesize that this curriculum
will encourage learning partial point cloud embeddings
and local descriptors, which will benefit online
recognition. Furthermore, similar to [5], we apply point
cloud filters to cope with the differences between tactile
and visual representation.
Figure 3 depicts the whole procedure. The resolution of
the difference in representations is tackled with preprocessing
of point clouds to a unified representation, as proposed
in [5], which used the same point cloud filters on both visual
and tactile data. Different from [5], we incorporated a DL network,
PointNet [10], to extract meaningful shape features and
make predictions using the network's multilayer-perceptron
output layers. We trained PointNet with the augmented vision
dataset V to learn the model parameters then used in the
prediction stage, where PointNet takes filtered tactile point
clouds T to infer the corresponding objects from O. This
augmentation, discussed in the " CL " section, is a part of the
broader CL strategy to recognize partial representations.
PREPROCESSING OF DATA
To reduce the differences in spatial resolution and noise and
achieve a unified visuo-tactile point cloud representation, we
considered the approach taken by the authors in [5]. Their
approach unified the representations in two stages: 1) denoising
and reconstructing the surfaces using a moving leastsquares
(MLS) filter and 2) equalizing local densities using a
voxel filter. The results of these filtering operations are exemplified
in Figure 4.
(a)
(b)
(c)
FIGURE 2. The system is required to recognize partial point
clouds with increasing accuracy as the number of points in the
point cloud increases. (a) 10% of points, (b) 40% of points, and
(c) 70% of points.
MLS FILTERING
First, the MLS filter explored in [17] was used to promote
local smoothness and reconstruct model surfaces. This process
has the observed effect of removing noise perpendicular
to the surface of the objects, hence sharpening planar
Training
RGB-D
Camera Sensor
3D Frames
Raw Visual
Samples
MLS
and Voxel
Filtering
Filtered
Visual Samples
Augmentation of
Vision Dataset
PointNet Training
Prediction
Trained
Model
Parameters
Robot Setup
Taxel Data
Raw
Tactile Samples
MLS
and Voxel
Filtering
Filtered
Tactile Samples
Tactile
Point
Cloud Tj
Evaluation
Object
Prediction Ol
FIGURE 3. The system architecture. The top row depicts the training stages. A vision point cloud generated from an RGB-D camera is
filtered using both moving least-squares (MLS) and voxel filters to reduce noise and equalize the spatial density of points. The vision
dataset V is augmented by applying transforms to each point cloud Vi, as discussed in the " One-Shot Learning-Data Augmentation "
section and used to train a PointNet. The bottom row depicts the prediction phase, where tactile samples are sensed and (optionally)
filtered. PointNet learns to approximate the mapping cT, forming from each tactile point cloud Tj a prediction of the object instance Ol
t
it
belongs to.
72 IEEE ROBOTICS & AUTOMATION MAGAZINE SEPTEMBER 2023
"

IEEE Robotics & Automation Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2023