IEEE Robotics & Automation Magazine - December 2022 - 42

The first identified gap in
robotic scene perception
is the missing usage of
multiple sensor modalities
as the input source.
red-green-blue cameras and 2D laser scanners set the entry
barrier in robotic developments, e.g., for student or hobby
projects, relatively low. Investigations mainly focus on a deep
learning-based detection of common objects in images or on
mapping the environment using SLAM. 2D methods in
object detection benefit from matured research in CV. It
reached high accuracy due to challenges, such as the Pascal
Visual Object Classes
Challenge [97] and the
Large Scale Visual Recognition
Challenge [98], by
public datasets of labeled
data, just like the Common
Objects in Context
(COCO) dataset [99]. For
the application on robots,
these 2D image-based
approaches must be transferred
to time-continuous
3D processing merged
with SLAM techniques providing the ego-motion of the robot
to reconstruct the environment.
Only comparatively new research focuses on 3D multiframe
scene recognition solving the high computational
performance with mobile graphic accelerators [49], [100],
[101]. However, both the representation and the high-level
interpretation of scene knowledge are not focused. Indeed,
as described by Neisser [3], solely the knowledge that was
recognized can be represented or interpreted. The focus on
sensor-close recognition in combination with visual and
mobile robotic challenges, such as changing visual appearance,
as well as the limited computation resources,
explains this observation. These aspects also explain why
research on the representation and interpretation of scene
knowledge is comparatively rare. As a result, robotic scene
perception in real-world applications is focused on a concrete
use case utilizing highly optimized bottom-up perception
pipelines with narrow functionality. For instance,
industrial environments have adapted to the perception
capabilities of robots.
Herewith, robots are enabled to fulfill narrow perception
tasks with high accuracy. In particular, deep learning-based
recognition systems are often used as monolithic black box
systems, being hard to combine efficiently without probing
deeply into a technical level. Research articles mostly bury
these challenges by describing specific techniques [21].
However, robots in everyday environments need to fulfill
multiple recognition tasks simultaneously as a requirement
for complex and extensive behaviors. Open source frameworks
such as the Robot Operating System (ROS) [102]
provide, thanks to its large community, many tutorials as
well as open source basic functionalities that are suitable for
creating a powerful overall performance based on single
software pieces. Standardized communications between various
software components enable robots to use, fuse, and
analyze multiple data streams.
42 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
However, on the one hand, paralyzing multiple narrow
perception pipelines is challenging regarding performance.
On the other hand, splitting pipelines into multiple steps, to
reuse, e.g., preattentive information, is not easy when using
monolithic black box models. Nevertheless, few researchers
worked on integrated top-down approaches that aim to perceive
the scene as a whole by combining multiple methods.
The Holistic Scope of the Latest
Integrated Approaches
There are a few approaches in research that integrate multiple
scene perception tasks into a single framework. In contrast to
the narrow perception techniques, they achieve a more holistic
understanding of the scene. These approaches simultaneously
recognize environmental information over a long time.
Storing these data in a known structure offers a new potential
for a more complex spatial and temporal interpretation of the
scene. We looked deeper into these approaches to extract how
much they cover a holistic scene understanding. In the following,
a comparison of the perceptual capabilities should
answer this question. As criteria, we select research approaches
covering more than a tabletop scene; reconstruct in real
time; and represent scene data by multiple types of instances,
such as semantics. However, each approach set a different
focus starting at sensor-close processing, such as on paralyzing,
fusing, and handling of dynamics going to ontologybased
reasoning.
Table 1 shows the perceptual properties of these
approaches, divided into the three steps of the perception
transfer presented in the " Transferring Human Scene Perception
to Mobile Robots " section. If we could not find the
details to a criterion, we marked it either as not available
(N\A) or not specified (NS). The approaches of Table 1 use a
3D camera as sensory input providing the color and depth
information of its FOV. Additionally, some approaches use
an IMU to support visual odometry for a more precise estimate
of the ego-motion, e.g., needed for drones and
wheeled-based robots. Based on the sensory input stream,
the presented approaches combine several recognition techniques
to reconstruct a virtual scene model. However, they
cover the recognition differently.
KnowRob [65], a knowledge representation and reasoning
framework, solely offers an interface for individual visual recognition
systems. Wyatt et al. [67] limit the reconstruction to
a metric map without recognizing semantics by vision sensors.
Its scene recognition is trained from visual properties by
a human tutor using supervised learning. Similarly, the
SOMA framework restricts the reconstruction to a metric
map but enhances objects by a CNN for color image-based
object detection. SOMA aims at understanding changes in
everyday environments by perceiving geometries and semantics.
Alternatively, the approach of Suchan et al. [103] enhances
the metric map by detecting walls, which are used for a
clustering algorithm to generate a floor plan. The other
approaches investigate further into a fully 3D metric-semantic
reconstruction of the scene.
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
