IEEE Robotics & Automation Magazine - June 2020 - 71

tactile image was collected. The perturbations took uniform
random values within ranges typical of movements that may
advance the sensor without damaging it, in all pose components except the vertical. Ranges for the target poses and the
unlabeled perturbations were chosen according to how much
the sensor could move to remain safely in contact without
causing damage.
Model Training
As is common practice in NN regression problems, the network weights and biases were optimized during training by
minimizing the mean square error (MSE) between the predicted outputs and target labels. The loss components were
weighted by 1/maximum2 for the parameters in Table 1
(e.g., depth: 1/52; roll: 1/152; and yaw: 1/452) to give peak
losses of order one. Several different regularization
approaches were used to avoid overfitting the training data,
including constraining the number and size of the hidden
layers and using early stopping during training. These
choices were part of the hyperparameter optimization process described in the following.
All PoseNet models were trained using the Adam adaptive
learning rate optimizer (initial learning rate: 10 -4; decay coefficient: 10 -6). This optimizer consistently gave good solutions
and seemed to converge more quickly than other types. Prior
to training, all weights were initialized to small random values. Early stopping was incorporated to terminate the optimization process, where the MSE loss was computed for a
separate validation data set and the process halted when
there was no further improvement through 10 epochs. Training and optimization of the deep NNs were implemented in
the Keras library on a Titan Xp graphics processing unit
(GPU) (12-Gb memory) hosted on a Windows 10 PC. A
training run typically takes 5-20 min, depending on the size
of the network.
Hyperparameter Optimization
Bayesian optimization was employed to give a systematic
approach to setting the network and training hyperparameters. The PoseNet models were optimized using 300 cost
function evaluations, incorporating 50 random startup evaluations, from the MSE loss on the validation data set after early
stopping. Bayesian optimization is a sequential, derivativefree, black-box method that is well suited to noisy and expensive-to-evaluate cost functions. It incrementally constructs an
acquisition function that directs cost function sampling to
areas where improvement is likely. We used a Python implementation of Bayesian optimization [HyperOpt (http://
hyperopt.github.io/hyperopt/], which constructs a generative,
nonparametric model of an acquisition function, the TreeStructured Parzen Estimator. Overall, the optimization and
training processes took 1-2 days on our hardware.
The model configurations were constrained to lie within a
search space bounded by parameter ranges that took account
of the available computing resources and reasonable values of
the hyperparameters (Table 2). For example, given the 128 ×

128 dimension of the input tactile image and the 2 × 2 maxpooling architecture of the network, the number of convolutional layers N conv # 5, and computing resources constrained
the number of filters per layer N filter # 512 and batch size
# 256. We emphasize that it was not obvious a priori where

τz
θnz

τz

θny

θny
θnx

θnx
τx

(a)

(b)

Figure 5. The pose parameters for (a) the 3D surface and (b)
the 3D edge. The surface has three parameters: depth, roll, and
pitch; the edge has five parameters: x-horizontal, depth, roll,
pitch, and yaw. The labels are in axis-angle coordinates (x, in).

Table 1. The training data pose parameter
ranges.
Labeled

Unlabeled

Parameter

3D Surface

3D Edge

Perturbation

x-horizontal

[−5, 5] mm

y-horizontal

[−5, 5] mm

Depth

[−5, −1] mm

0 mm

Roll

[−15, 15]°

[−5, 5]°

Pitch

[−15, 15]°

[−5, 5]°

Yaw

[−45, 45]°

[−5, 5]°

-1.1 mm -2° 0.5° -4.2 mm 13.9°-1.1° -4.4 mm -0.7°8.3°

-1.1 mm -1.6° 1.2° -4.2 mm 14.8°-1.6° -4.4 mm 0°

7.6°

Figure 6. The example preprocessed tactile images for the 3D
surface and their pose labels (depth, roll, and pitch). The tactile
images were selected to have similar labels for each column (equal
depth; roll and pitch within 1º). Differences between rows are due
to the unlabeled random perturbations in the pose, and are most
apparent for the strongest/deepest contacts (right column).

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

http://hyperopt.github.io/hyperopt/ http://hyperopt.github.io/hyperopt/

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020