Instrumentation & Measurement Magazine 24-3 - 85

Biologically Inspired Vision and
Touch Sensing to Optimize 3D
Object Representation
and Recognition
Ghazal Rouhafzay, Ana-Maria Cretu, and Pierre Payeur

3

D representation and recognition of objects are two
pivotal steps for autonomous robots to safely explore
and interact with an unknown environment and manipulate objects. 3D modeling can be beneficial in different
robotic applications such as object grasping, pose estimation,
robot navigation and localization. Real-time data acquisition
and accurate object representation are essential in the context
of such practical applications. On the other hand, the recognition of the objects in an environment is indispensable for
situational awareness and for enabling the robot to interact effectively with complex environments.
Robot vision can be considered as the most informative and
reliable sensing modality in autonomous robots. Nevertheless,
vision fails to work properly in a number of situations including low light environments, cases where an object is occluded
or is out of the camera's field of view, and situations in which
objects are not visually distinguishable. Tactile sensing, as an
indispensable element of dexterous robotic manipulation, can
be efficiently integrated with other sensory modalities, in particular with vision, to increase the reliability of an autonomous
robot. It makes available a wide range of information on objects
including surface properties such as roughness, texture, vibration, temperature, local shape, etc., all important features that
can contribute to better identifying an object. Moreover, a combined use of vision and touch in humans was demonstrated to
facilitate manipulation, grasping and handling of objects, and
could therefore be exploited to increase the efficiency of autonomous robots in a variety of tasks. However, visuo-tactile
integration and the creation of efficient computation methods
to help a robot successfully recognize and manipulate the objects it is interacting with remains a challenging issue.
A huge research effort has been invested in the literature
to efficiently integrate the two sensing modalities. Nevertheless, all currently published works tackling visuo-haptic
interaction only use visual data to increase the spatial resolution of tactile data, to resolve conflict situations, such as cases
where the tactile information is faulty, or conjunctly use tactile and visual data to recognize objects. Considering the fact
that the acquisition and processing of tactile data itself is a
May 2021	

time-consuming task, such approaches for visuo-tactile integration are associated with a high computational cost, thus
making them very difficult, if not impossible, to use in realtime interaction scenarios. Alternatively, the sophisticated
cognitive skills of the human brain and its patterns of natural
intelligence have encouraged scientists to develop biologically
inspired computation techniques, bringing automatic processing capabilities to computers and robots.
Referring to biological research, we can draw three main
conclusions about the interaction and collaboration of visual
and haptic sensory modalities: 1) Tactile salient features also
attract visual attention to their location [1]; 2) a combined use
of vision and touch works more efficiently compared to cases
where vision and touch are exploited separately [2]; and 3) visual and tactile object recognition rely on similar processes in
terms of categorization, recognition and representation [2].
These conclusions suggest that visuo-tactile integration is a
promising solution to optimize the process of object modelling.
Moreover, visual data, in the form of salient regions acquired
by a model of visual attention (according to 1) can be employed
to guide the process of tactile data acquisition. Furthermore, visuo-tactile integration can be performed (according to 2 and 3)
at a higher (perception) level based on similarities between the
two sensing modalities. Since collecting large datasets of tactile
data for training a model is a much more complex task compared to visual data, it is expected that a transfer of learning
from vision to touch can both enhance the performance of tactile object recognition and amalgamate visual and tactile data
processing units in robots. This paper presents research initiatives performed by the authors to validate these biologically
inspired assumptions and efficiently merge measurements
from different instrumentation technologies in a framework to
operate in the context of practical robotic tasks that involve 3D
object representation and recognition.

Techniques for Visual and Tactile Data
Acquisition
As illustrated in Fig. 1, the framework makes use of visual
and tactile data acquired over the surface of objects. A variety

IEEE Instrumentation & Measurement Magazine	85
1094-6969/21/$25.00©2021IEEE
Instrumentation & Measurement Magazine 24-3

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-3
