IEEE Robotics & Automation Magazine - December 2022 - 39

perception to recognize objects [25], map the environment
[26], [27], or avoid obstacles [28] and haptic perception to
actively touch objects [16]. The linking of action to perceive
the information of interest is known as interactive
perception, which is a common human exploration strategy.
For instance, if humans cannot recognize an object by
vision and touch, they will take different interactions to
obtain information from other sensory channels [25].
The usage of smelling and tasting for robots is not popular.
One rare example presents a navigation approach using
smell for odor source localization [29]. A different niche
develops a tasting sense using IR spectroscopic technologies
or chemical sensors to discriminate food [30], [31]. Driven
by computer vision (CV), 2D image processing is the fundamental
pillar for robot recognition tasks. However, 3D
methods for scene and object reconstruction gained importance
for robots. Various surveys provide an overview of the
state-of-the-art robot recognition tasks [32], [33]. Established
robotics approaches focus on specific recognition
subareas. The preattentive recognition of primitives comprises
methods on normal estimation [34], [35], segmentation
[36], [37], edge [38], and simple shape (such as planes
and cylinders) [39] detection. The methods usually make
use of the entire sensory input. However, robots prefilter or
scale the data to achieve the appropriate results on their
capabilities. For instance, image-based feature recognition
can process high-resolution images by making use of image
pyramids [40]. Herewith, the extracted features of a lower
image resolution deliver discrete regions of interest by keeping
the accuracy of the original scale.
Postattentive processing comprises higher-level recognition
capabilities based on preattentive processing. Examples
are extracting the semantic by (re)-detection of objects [41],
[42], humans [43], [44], and places [45]; 3D reconstruction of
the scene by Simultaneous Localization and Mapping
(SLAM) [32], [46], [47]; or enriching metric information by
semantics, known as semantic segmentation [33], [48], [49].
Equal to humans, robots use preprocessed attention
areas within postattentive processing. For instance, featurebased
SLAM methods use sparse feature points to postattentively
register a set of sensor data and to detect loop
closures to compensate for the mapping drift [46], [47].
Besides, this example shows how humans and robots use
different sensory sources to achieve highly accurate localization
within the scene. Humans use walked steps combined
with eye information for perceiving ego-motion,
which improves localization. For illustration, imagine walking
a distance with closed eyes. The motion drift sooner or
later leads to a loss of localization. Therefore, a series of
interconnected spatial-visual features provides the absolute
localization within the known scene [50].
Equivalent to humans, the latest robotic SLAM approaches
provide similar techniques. Wheel odometry, visual
odometry, or an inertial measurement unit (IMU) provides
the ego-motion. Laser scanners or camera-based
methods simultaneously estimate the pose of landmarks to
compensate for the ego-motion drift. Also, some approaches
use visual landmarks for place redetection to close the trajectory
loop while mapping [46]. The usage of high-level features
enables robots to add semantics to metric information.
Knowing the semantics of an area in space helps robots to
interpret the scene. It allows, e.g., the exclusion of dynamic
objects like people while mapping.
A special challenge of robot perception is the recognition
of scene information from different locations and times. For
instance, a robot recognizes an object in the scene and simultaneously
tracks the object in its field of view (FOV) as long as
it is visible. While previous research proposed using a Kalman
filter [51] or particle filter [52], the latest research utilizes deep
learning-based tracking, such as with a convolutional neural
network (CNN) [53], [54] or a Vision Transformer (ViT)
[55]. ViT, coming from natural language processing (NLP),
splits images into fixed-size patches that gained popularity
due to their superior performance on continuous data
streams, needed for mobile robots [56]. Since the object and
the robot could move, occlusions, truncation, or invisibility
due to sensor noise (as mentioned previously) must be handled.
When the same object appears again, a reidentification
(Re-ID) to reallocate the ID is beneficial to better understand
the scene. Modern approaches of Re-ID have been proposed
that are similar to tracking the usage of a CNN [57], [58], ViT
[59], [60], or end-to-end approaches [61].
Knowledge Representation
The second step of the perception process represents environmental
information. A knowledge base manages recognized
information from different sources and levels of
abstraction, times, and places in a centralized and ordered
structure. This structure includes understandable information
about the scene. The function of the knowledge representation
within the human brain has been a controversial
topic since the so-called gestalt theory. Modern attempts
such as Wagemans and Kimchi [24] or Hommel et al. [62]
reveal spatial layouts, organized hierarchically, that represent
the human perception memory. A relationshipfocused
multilevel hierarchical structure of parts represents
environmental information.
The transfer of the main functionalities of human
knowledge representation to robots requires a complex
memory structure focused on flexibility. The knowledge
representation must be capable of merging observations
and interpretations from different sources and times. For
instance, recognized information, such as shape, texture,
posture, state, probabilities, and trajectory (compare the
" The Recognition of Information " section), must be managed
in real time within the knowledge base. This issue
sets high requirements for the underlying knowledge base
as every piece of environmental information needs a
known structured representation. Furthermore, the
knowledge base includes initial and postprocessed knowledge.
Robots usually store and represent the scene knowledge
in a database [63], [64], [65], allowing them to deploy
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
39
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
