IEEE Robotics & Automation Magazine - September 2023 - 71

closely resemble data gathered from a series of tactile interactions
with an object.
The main contributions of this article are
■ a data-driven pipeline capable of recognizing objects from
partial tactile observations, and the experimental evaluation
of this pipeline on our collected dataset.
■ the CMR of objects represented by points distributed over
a nonplanar manifold. This represents a challenge because,
as explained in the " Problem Definition " section, it is not
always possible to obtain a complete tactile model of nonplanar
objects. Therefore, an architecture capable of performing
predictions based on partial information is
required in this scenario.
■ a CL pipeline that encourages task-specific descriptors.
The remainder of this work is structured as follows. An
overview of the problem statement and design requirements
is given in the " Problem Definition " section. The " Methodology "
section proposes the system architecture and presents
a curriculum training procedure. The " Experiments " section
presents details on the experimental setup, datasets, and data
processing. Finally, the " Results and Discussion " and " Conclusion "
sections conclude the article.
PROBLEM DEFINITION
Let O:{ |, ,}
Oll f=1 L be a set of L known objects. The single
object OOl ! is represented using two distinct modalities:
visual and tactile. In the specific case, visual information is
acquired using a red, green, blue-depth (RGB-D) camera;
conversely, tactile data are collected using a tactile-sensing
array integrated on a robot's end effector. Both visual and
tactile information can be represented as a point cloud,
assuming the position of each tactile element is known with
respect to a common reference frame. Therefore, when the
robot's end effector gets in contact with the object, it is possible
to associate a small point cloud (whose size is related
to the number of sensors composing the tactile array) to a
specific position in the space. This assumption is not specific
for the tactile-sensing technology used in this article (see
the " CySkin and Collection of Tactile Data " section);
indeed, it holds for any tactile sensor composed of a set of
independent transducers.
More formally, let us denote with
V Vi M1
ii=
Tj NT n==f
1
:{ |,i f=
|| }Vm the visual dataset containing the visual point
clouds for all objects O Ol ! and, in the same fashion,
T jj j
:{ |,|| } containing the tactile point
clouds. Here, V and T represent the objects O as sensed
through a vision and a tactile sensor, respectively, where
each point in the point clouds is expressed in Cartesian coordinates.
Let us also define two functions, cV( · ) and cT( · ),
which, taken visual and tactile point clouds, respectively,
return the object instance Ol.
This article addresses the development of a data-driven
model that can learn to recognize the specific object OOl !
from a partial tactile point cloud
j
mate the mapping cT and perform an estimate O ,lt
TT ,1 j i.e., learn to approxifrom
visual
point clouds V and a known mapping cV. Furthermore, we
(d)
(e)
(f)
FIGURE 1. The top row shows (a) the raw visual point clouds of a
beer can, (b) a Rubik's cube, and (c) tape. On the bottom row, (d)-
(f) represent the corresponding tactile point clouds. Plots (b) and
(e) highlight that the tactile point clouds failed to capture edges.
Further, some surfaces captured in the visual clouds are not
present in the tactile clouds, such as the inner surface of the tape
reel, plotted in (c) and (f). The recognition system will need to be
robust to such missing edges, manifolds, and surfaces in addition
to varying densities and qualities of the point clouds.
SEPTEMBER 2023 IEEE ROBOTICS & AUTOMATION MAGAZINE
71
seek a solution that can operate under low amounts of training
data
V , and specifically, one-shot learning.
To summarize, we design a system able to perform a tactile-based
recognition under the following criteria:
■ a partial representation of point clouds
■ differences in sensor characteristics between visual and
tactile point clouds
■ few training samples per object.
Figure 1 presents examples of point clouds generated from
visual and tactile data. The image shows the extent of the differences
between the two representations: although the visual and
tactile modalities both captured smooth faces and the overall
shapes of objects well, the tactile point clouds (d)-(f) achieve
lower surface coverage than vision (a)-(c). Furthermore, when
considering nonplanar objects, a full tactile representation cannot
be always reconstructed as some parts are not reachable
(e.g., the inner surface of the tape reel in Figure 1(f). We remark
that the differences shown in Figure 1 are not specific to the
tactile-sensing technology described in the " CySkin and Collection
of Tactile Data " section and used in this article. Indeed,
as there are no standards at the hardware level [16], the spatial
resolution and distribution of tactile-sensing elements change
depending on the adopted technology. This article directly
addresses the modality differences: although point clouds can
represent both visual and tactile data, in general, their differences,
as shown Figure 1, can affect the overall performances
of the recognition system.
Furthermore, as the tactile-based exploration of the whole
object is a time-consuming process, we aim at recognizing the
object from partial information by defining a process where
recognition accuracy improves as more parts of the objects are
sensed. Figure 2 depicts an example of a partial tactile point
cloud. Although beyond the scope of this article, this will enable
(a)
(b)
(c)
IEEE Robotics & Automation Magazine - September 2023

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2023
