IEEE Robotics & Automation Magazine - June 2020 - 68

CNN with 241 tactile inputs and 71 motor angles, currents,
and force/torque readings. The authors also observed that
they were likely the first to produce research in this area
because of the difficulty of gathering many samples of highdimensional data with tactile sensors.
During the next couple of years (2015-2016), a handful of
studies followed. These applied deep learning to pressure-sensitive tactile arrays on the fingertips of robot hands using
CNNs for tactile shape recognition [4], material texture classification [5], and slip identification [6] as well as a recurrent
long short-term memory network for classifying held objects
[7]. These studies used spatial and temporal data from their
tactile arrays, which in combination give a high dimensional
input suitable for deep learning. Related approaches have continued since, with a variety of taxel-based sensors and hands.
In 2016, a related area that combined tactile and visual
data branched off from this early work. This was first applied
to haptic adjectives from visual images and a single BioTac
sensor [8], and the authors observed that NNs serve as a natural unifying framework for multimodal signal fusion. Soon
after, visuo-tactile sensing was applied to object classification
and grasp planning with a red-green-blue (RGB) depth camera and tactile arrays on a robot hand [9]. This area has continued to develop since, with multiple studies connecting look
and feel [2], [10], [11], progressing more recently to deeplearning methods that can transform between [12] or match
[13] visual and tactile data.

Optical Tactile Sensing for Deep Learning
During the past few years (2017-2019), the adoption of optical tactile sensors by several research groups has found a natural synergy with deep learning (Figures 2 and 3). These
optical tactile sensors use an internal camera to image the
deformation of a compliant sensing surface in contact with a
physical stimulus. Although optical methods have been considered promising for tactile sensing since the 1980s [16], in
practice, the field had been dominated by various means of
electromechanical transduction. More recently, however, a
coherent body of research on deep learning for robot touch
has used optical transduction to make progress on previously
intractable problems, as typified by most of the recent progress in visuo-tactile sensing [2], [10]-[13].
The first application of deep learning to an optical tactile
sensor was for shape-independent hardness estimation using
the GelSight [17] [Figure 2(b) and (d)]. Tactile images (960 ×
720 pixels) were fed through a pretrained deep CNN into a
recurrent (LTSM) network to predict the Shore hardness of the
contacted object; training was with 7,000 video sequences (five
frames each) across various object shapes and levels of hardness. A major benefit was that the hardness estimation was
insensitive to nuisance variables, such as the object shape and
loading of the contact, that have a complicated influence on the
sensor output and would be very difficult to model otherwise.
Since that first study, the GelSight with deep learning has been
used for various tactile robotic problems, including most of the

(a)

(b)

Hemisphere

Edge

(c)

Hemisphere

Cylinder

(d)

Figure 2. The (a) BRL TacTip biomimetic optical tactile fingertip and (b) GelSight optical tactile sensor. The bottom panels show (c)
TacTip and (d) GelSight tactile images from manual contact against a few test stimuli. The GelSight tactile images are from [2], and
the TacTip tactile images were generated for this article.

68

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

JUNE 2020
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
