IEEE Robotics & Automation Magazine - December 2021 - 49

having five convolutional layers and one dense layer (Table S3
in " Training Data, Control, and PoseNet Parameters " ).
The accuracies of these surface and edge PoseNets are
reported in Table 1. The horizontal and yaw pose accuracies
were much improved over previous results [11]; other pose
parameters have similar or slightly poorer accuracies, which
we attribute to unimportant differences in the sensor and its
placement during data collection. The key point is that this
method gives accurate pose estimates that are insensitive to
shear as the sensor interacts with an object.
Tactile Robotic System
Tactile Sensor
This study involves the use of a soft biomimetic optical tactile
sensor from the BRL TacTip family [12], [27]. The use of 3D
printing enables the designs to be highly customizable,
encompassing stand-alone sensors and fingertips of robot
hands with various shapes and sizes [12]. Printing multiple
materials in one piece allows the combination of a rigid casing
(VeroWhite) with a soft sensing surface (TangoBlack). The
version used here has a soft tactile dome (40 mm in diameter)
with 127 internal pins tipped by printed white markers that
function as tactile elements.
An important aspect of the soft biomimetic design is that
the sensor is highly sensitive to shear deformation, which
induces a global movement of all pins along the direction of
lateral strain in the sensing surface. While this is useful for
detecting phenomena such as slip [28], it causes issues for tactile
servoing when sliding over object surfaces [10], [25].
The problem is that the tactile image depends not only on
the pose of the object but also on the prior transverse movement
of the sensor. As shown in previous work [10], this
causes the tactile servoing to fail under sliding motion unless
the pose can be inferred accurately. Here, we mimic the effect
of this transverse motion during training data collection so
that the PoseNet is trained to be insensitive to sensor motion.
Tactile Robot
The TacTip optical tactile sensor is mounted as an end effector
on an industrial robot arm (Figure 4), which was also used in
earlier studies of control using this tactile sensor [6], [10], [25].
We use an IRB 120 (ABB Robotics): a fairly small, six-axis
industrial robot arm (reach of 0.58 m, 25-kg weight, and 3-kg
payload) with accurate positioning (0.01-mm accuracy and
+ 1-s cycle time). The TacTip base is bolted onto a mounting
plate attached to the rotating (wrist) section of the arm.
The closed-loop control of this robot is limited to long
L100 ms since the IRB 120 is aimed at applicalatencies
()
tions where its end effector passes through predefined waypoint
poses. Therefore, we control the arm in discrete updates
of the end-effector pose, with the industrial robot controller
(IRC5) calculating the motion between poses. While this
setup is primitive compared with many research robots, e.g.,
cobots that react to humans in real time, it does provide a
basic platform sufficient for investigating PBTS control.
●
Software Infrastructure
The software and hardware are integrated with four main
components written mainly in Python:
●
●
Tactile images are collected using the OpenCV library and
then preprocessed and saved for training or used directly
for prediction.
These images are cropped, thresholded, and subsampled in
OpenCV to give 128128-# pixel images, which are then
passed to the deep learning models implemented using the
Keras deep learning library.
●
The resulting predictions are passed to control software in
Python [with some visualizations using the MATLAB
engine application programming interface (API)].
Finally, the computed control signal is sent to the robot
arm via the RAPID API.
The training and optimization of the deep neural networks
was implemented on a Titan Xp GPU hosted on a
Windows 10 PC. A training run typically takes about 10 min.
The learned model is then copied to the robot control PC,
which, in this case, was a Windows 10 PC with limited GPU
hardware. Even so, the PoseNet models typically made
predictions in about 50 ms. The longest delay within the
cycle was the latency between the control PC and the robot
arm (typically 100 ms).
Table 1. The PoseNet accuracy:
the MAE of predictions.
Parameter Surface (MAE) Range Edge (MAE) Range
Horizontal, x -
-
Vertical, z 0.1 mm
Roll, a
Pitch, b
Yaw, c
0.4°
0.5°
-
0.3 mm
4 mm 0.2 mm
30°
30°
-
1.2°
2.4°
4.1°
10 mm
4 mm
30°
30°
90°
Tests were performed over an independent set of 2,000 samples. MAE:
mean absolute error.
(a)
(b)
Figure 4. The tactile robotic system, showing (a) a close-up of
the tactile sensor and (b) a view of the sensor arm system next
to some test objects.
DECEMBER 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
49
IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021
