IEEE Robotics & Automation Magazine - June 2020 - 70

These processed features are combined linearly at the output layer to give N out pose components. Unlike previous
work in deep learning for tactile sensing, we did not handtune these network parameters or rely on values from
other studies. Instead, we treated them as unknown hyperparameters to be optimized in the training process. This
approach has been used widely elsewhere in deep learning,
but, to the best of our knowledge, this is the first time it
was applied to touch.
Other hyperparameters of the PoseNet architecture and
training process that were optimized include the activation
function used in the hidden layers [rectified linear unit
(ReLU) or exponential linear unit (ELU)] and regularization
parameters to avoid overfitting the training data, encompassing batch normalization, dropout, and L1/L2 regularization. Where batch normalization was used, it was inserted
between the convolution operation and activation function
in all convolutional hidden layers. Where dropout was
employed, it was applied to the inputs of all dense hidden
layers and the output layer, before the dense matrix multiplication. Where incorporated, L1 or L2 regularization was
applied to the weights in all dense hidden and output layers.
The reason for including more than one type of regularization approach is that the methods tend to work in complementary ways: Batch normalization and dropout inject
noise into the training process, helping to prevent the complex coadaptation of features, whereas L2 regularization
encourages smooth mappings, and L1 regularization
encourages sparse models.
In this article, we considered two distinct types of PoseNet:
one for a 3D surface and another for a 3D edge (Figure 4). In
addition to differences in their trained parameters, the networks have different outputs. The network for the 3D surface
has three outputs: the depth, roll, and pitch of the contact; the
network for the 3D edge has five outputs: horizontal distance,
depth, roll, pitch, and yaw. These parameters are sufficient to
describe the pose of a tactile sensor in contact with a surface
or contour by defining the local pose relative to a plane or line
on a tangent to those objects.

Training Data and Network Optimization
One contribution of this article is to identify some subtleties
in the data collection and hyperparameter optimization process when developing deep NNs for robot touch. Robot touch
senses via contact, so the manner of contact affects the tactile
data. To predict the pose accurately, the trained network
should be insensitive to how the sensor has reached its current pose; for example, the motion of the sensor sliding across
a surface or along an edge (Figure 1). Here, we included representative examples of these motions as unlabeled perturbations of the training data. However, these perturbations make
it more difficult to predict the pose, to the extent that improperly tuned hyperparameters gave extremely suboptimal
results. For a systematic approach, we used automatic-tuning
methods, specifically Bayesian optimization.
Training Data Collection
Three data sets were collected for each of the two objects considered here (a flat surface and straight edge; Figure 5), giving
independent training, validation, and testing data. Each set
had more than 2,000 contacts, with the poses sampled randomly from uniform distributions within their allowed ranges (Table 1). Separate training and validation sets were used to
mitigate the potential data set shift, rather than using a randomized split. The data consisted of single tactile images
cropped and subsampled to a 128 × 128-pixel region that
viewed all tactile elements of the sensor surface. These images
were then preprocessed with an adaptive threshold to produce binary (black/white) inputs to the NN (see the examples
in Figure 6), reducing the sensitivity to changes of the lighting
inside the sensor.
We found it crucial to mimic the effect of motion on the
tactile data while collecting the training data because motiondependent shear affects measurements from optical tactile
sensors, such as the TacTip. Each sample of data had a random labeled pose and a random unlabeled shear perturbation
(the ranges are given in Table 1). First, the sensor was brought
into contact at the labeled target pose minus the unlabeled
perturbation, then moved to the target pose; at that point, the

Nconv Convolutional Hidden Layers

Ndense Dense Hidden Layers

128

Binary
Tactile Image

Object Pose
Parameters

3×3
Nout

128
Nfilters

Nunits

Nfilters
Figure 4. The PoseNet CNN. A binary sensor image is processed by a sequence of N conv convolutional layers, each containing N filters
filters. The resulting convolutional features then pass through a sequence of N dense fully connected hidden layers, each containing
N units processing units, to produce a high-level feature vector. The high-level features are combined by a linear output layer to give
N out continuous-valued object pose parameters.

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020