IEEE Robotics & Automation Magazine - June 2020 - 69

visuo-tactile studies mentioned previously [2], [10]-[13]; control of rolling motion [18]; and grasp stability [19] and grip
readjustment [15] on two-digit grippers (Figure 3).
Recently, deep learning has been found highly suited to
another type of optical tactile sensor: the BRL tactile fingertip,
known as the TacTip (Figure 2). The first study considered
robust edge perception for exploring objects with contour, following [1]. Tactile images (128 × 128 pixels) were fed into a
CNN to predict the 2D edge pose; training was with 2,000
videos (five frames each) across a range of edge angles and
positions. The challenge was to make the pose estimation
insensitive to motion-dependent shear during sliding, which
was addressed by hand-tuning the network architecture to
have a stack of input convolutional layers that learned broader
features across the tactile image. The TacTip has since been
used with deep learning to identify benchmark objects and
predict the grasp success of a three-finger robot hand [14]
[Figure 3(a)] as well as to extract features for predicting the
shape, 2D edge pose, and force with a convolutional autoencoder [20]. These studies used standard CNN architectures to
make predictions that are insensitive to nuisance variables,
such as the unknown object pose after automated grasping.
Why Optical Tactile Sensing?
Given our opening comments about the barriers to applying
deep learning to touch, why has a large proportion of research
in that area used optical tactile sensing? In our view, the GelSight and TacTip tactile sensors share some commonalities
that suit deep learning; for example, both use an internal camera and internal light sources to image the skin deformation.
That said, there are also significant differences that have
implications for future development and use. Historically,
both sensors were invented before the present revolution in
AI. The first paper on the GelSight was in 2009, on retrographic sensing for the measurement of surface texture and
shape [21]; the first paper on the TacTip was also in 2009, on
the development of a tactile fingertip based on biologically
inspired edge encoding [22]. Since then, both sensors have
progressed in their designs and uses, but their operating principles have remained the same [23], [24].
The GelSight operates on the principle of measuring the
normal indentation of a surface based on the shading of light
reflected off the underside of that surface [21], [23] so that,
with multiple light sources, a depth map can be reconstructed
using a photometric stereo algorithm. The most common
design uses internal RGB lights arranged symmetrically
inside the sensor to give three sources of shading [Figure 2(b)
and (d)]. Although the original intention was to measure
indentation directly, when used with deep learning, the
unprocessed RGB images are fed into the input layer of a
CNN to extract tactile image features.
The TacTip operates by measuring the deformation of a
surface from the shear displacement of markers on the
tips of 3D pins protruding beneath that surface [22], [24]
[Figure 2(b) and (d)]. This design is biomimetic because
human fingertips have an analogous structure in which

mechanoreceptors lie on papillae protruding between the epidermal and dermal skin layers. Originally, the pin tips were
detected and tracked to give tactile features, most commonly
as a time series of (x, y) displacements of the markers. However, when used with deep learning, the images are fed directly
as inputs to a CNN.
In our view, the GelSight and TacTip sensors are well suited for deep learning because they both produce tactile images
where every pixel gives information about surface deformation. For the GelSight, every pixel has RGB intensity readings
that relate to indentation. For the TacTip, every pixel can signal whether or not a pin has moved to that location. Moreover, in recent years, the GelSight design has been modified to
have markers inside the skin to indicate shear [23], which is
similar to the operation of the TacTip (but without its 3D pin
structure). Meanwhile, depth maps can be inferred indirectly from TacTip pin positions (for example, by using a
Voronoi transformation).
PoseNet: 3D-Pose Estimation From Touch
To illustrate the application of deep learning to robot touch,
we considered local pose estimation of a 3D surface in contact
with the TacTip optical tactile sensor. Pose estimation is a
fundamental capability of tactile sensing because knowledge
of the relative pose enables a tactile robot to control its interactions. In the following, we refer to this NN as a PoseNet
because it estimates the 3D pose from a tactile image.
Following previous work with optical tactile sensors [1],
we used a standard CNN architecture to generate and process
tactile features to predict the 3D pose (Figure 4). The tactile
image feeds forward through a sequence of N conv convolutional and max-pooling hidden layers, using N filters filters in
each layer and 3 × 3 kernels with unit stride (no padding).
The resulting tactile features then pass through further N dense
fully connected hidden layers, each with N units processing
units that compute a dense matrix multiplication plus bias.

(a)

(b)

Figure 3. The (a) TacTip optical tactile sensor integrated with
a robot hand and (b) GelSight optical tactile sensor integrated
with a gripper. The TacTip-enabled hand is the BRL Tactile Model
O [14]; the GelSight is integrated as the GelSlim sensor [15].

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

69
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
