IEEE Robotics & Automation Magazine - June 2020 - 67

through to the subcortical and cortical visual-processing
structures of the brain. In 2012, AlexNet won at the ImageNet
competition, sparking a revolution in computer vision. However, deep learning has spread quickly to other applications,
such as speech recognition, so this initial impetus to vision
does not explain the later disparity with touch.
In our view, the main barriers to applying deep learning to
touch are 1) a lack of cheap, robust, and easy-to-use artificial
tactile sensors, contrasting with modern cameras for computer
vision; 2) the difficulty of obtaining high-quality tactile data
due to a lack of public repositories and because a robot is usually needed to investigate the most interesting research questions; and 3) a lack of interest in the AI community for
applying deep learning to touch. The latter barrier seems
almost paradoxical, when most reports on AI aimed at policy
makers and the public are filled with pictures of humanoid
robots that will be useless in practice without functional hands.
In this article, we illustrate the application of deep learning
to robot touch by considering a basic yet fundamental capability: estimating the relative pose of part of an object in contact
with a fingertip. Tactile sensors can estimate the pose of the
region of the object being contacted by inverting the tactile
image into geometric features of the contact. However, finding
the relation between high-dimensional tactile images and the
low-dimensional pose is a challenge: Tactile sensors such as
our fingertips are soft and curved, so physical interaction
deforms the sensor in complex ways depending on the object
shape, contact forces, and contact history. In our view, this difficulty has confined the use of robot touch to very primitive
tasks compared with the fine motor capabilities of humans.
Accurate estimation of pose from touch will enable robots
to safely and precisely control their physical interactions. For
example, pose information can enable precise control of a fingertip sliding across complex objects, analogous to how
humans trace their fingers across novel objects to explore
shape. Previous work [1] demonstrated contour-following

around planar objects in 2D using deep learning applied
directly to the tactile images but required that the network be
carefully hand-tuned; otherwise, the pose estimate would fail
because the sensor sheared while sliding across the object.
Here, we adopt a different approach by collecting training
data that simulate the effect of shear and then using a blackbox Bayesian optimizer to select the network architecture
and other associated hyperparameters. In consequence, we
demonstrate controlled sliding motion across complex 3D
objects (Figure 1).
The main contributions of this research are to
1) show how deep learning can be used to train accurate
models to estimate 3D pose from tactile images
2) develop pose-estimation models that are insensitive to nuisance variables, such as motion-dependent shear, by directly incorporating the variables into the data collection
3) introduce a systematic approach to model selection, which
is needed for the most accurate models yet has not been
used before for touch.
We also take the opportunity to survey deep learning
applied to tactile robotics. In particular, we focus on optical
tactile sensors (Figure 2) that use an internal camera to image
skin deformation, since these sensors help bridge advances in
deep learning for vision and the new domain of touch.
Background
Deep Learning for Tactile Sensing
Initial applications of deep learning to artificial tactile sensing
were with taxel-based sensors, composed of discrete tactile
elements embedded in a skin. The first study, in 2014, focused
on tactile object recognition based on deep learning and
dropout [3] and using a four-fingered robot hand with pressure-sensitive capacitive tactile arrays on the palm and finger
joints/tips. Overall, the hands could recognize 20 objects held
in a variety of grasps at a success rate of roughly 90%, using a

Figure 1. The robot trajectories on a complex 3D surface and edge (a porcelain bust and a container top) using pose estimation while sliding
across the object.

JUNE 2020

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

*

67
IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020
