IEEE Robotics & Automation Magazine - December 2022 - 44

information within the layers. The knowledge representation
of SOMA comprises three layers inside a document-based
database: the observation layer, semantic layer, and interpretation
layer. The scene information is organized similarly to
Kimera with a spatial layout of hierarchical structures connected
by graphs. Suchan et al. [103] propose using ontologies
and formal characterization to represent knowledge by
space and motion. The knowledge representation of KnowRob
is the most powerful due to the usage of description
logic for representing and connecting knowledge within
multiple levels by the
Web Ontology Language
(OWL). The framework
links an inner world and
virtual, logical, and episodic
memories [65].
Based on the represented
knowledge, a few
approaches look deeper
into the interpretation.
For instance, the second
pillar of KnowRob is
knowledge-based reasonA
fundamental difference
between robots and
humans is the acquisition
of perceptional
understanding.
modality is affected, such as vision by illumination, color,
occlusion, and posture of objects [25] (see the " Recognition of
Information " section). The hardware design of robots allows
them an optimal sensory input due to flexible sensor choice,
amount, and alignment. On the one hand, this underlines the
statement of Premebida et al. [33] that robotic recognition
tasks, such as vision-based object classification, could deliver
higher performance than humans. However, on the other
hand, all the robotic approaches of the " The Holistic Scope of
the Latest Integrated Approaches " section limit the scene perception
to visual sensors, except for ego-motion sensors, as a
single modality. Although vision sensors provide the highest
information content of the scene, using a single modality is
insufficient for the situations mentioned previously (the
localization of acoustics; haptic feedback of obstacles; and
darkroom problem). Therefore, current integrated scene perception
approaches are not able to deploy different senses in
the way that humans do.
Different Perceptual Learning
A fundamental difference between robots and humans is the
ing to learn general knowledge. An object class-related affordance
extraction reasons what to do with objects [90]. Suchan
et al. [103] show by temporal human detection and object
recognition how to use reasoning to enhance the scene
understanding. The use of logical spatial relations between
instances (e.g., an object is located to the left of another
object) and the detection of human activities represent valuable
information about the scene.
The presented approaches indicate that they already perform
more than one perception process step on an advanced
level. Due to the different focus of every approach, none sufficiently
covers holistic scene perception. The degree of coverage
is difficult to quantify due to missing metrics in research.
However, the overview (see Table 1) offers a starting point to
identify gaps and potentials compared to human perception.
The Gap to Human Scene Perception
How well does robot scene perception mimic human-level
performance nowadays? Previous research compared the
perceptual performance of robots with children's age, such as
Szeliski [107], who claimed that the computer vision reached
the level of a two-year-old child. This comparison might not
be beneficial since robot perception is not quantifiable in
human terms as there is a specific rather than a broad perceptual
skill development of robots. Following this hypothesis,
this article details major gaps in robot perception in
everyday scenes.
The Nonusage of Sensory Modalities
The first identified gap in robotic scene perception is the
missing usage of multiple sensor modalities as the input
source. Humans use different senses, providing the opportunity
to fuse and rely on the optimal sensor since each sensor
44 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
acquisition of perceptional understanding. Perceptual learning
was first defined by Gibson [108] as " any relatively permanent
and consistent change in the perception of a
stimulus array, following practice or experience with this
array. " Thus, humans learn an individual perception without
initial knowledge that is constantly adjusted and influenced
by society and the environment. The learning enables
humans to achieve an optimal perception within the surrounding
since, in particular, high-level scene interpretations
depend on the subjective impression that is difficult to
generalize (see the duck-rabbit illusion [78]). In contrast to
humans, popular perceptual techniques of robots use rigid
learning strategies, which do not offer adaptation. It would
require retraining or parameter adjustment of the initially
deployed models. Moreover, these approaches provide neither
the flexibility for extension nor adaptation of the perception
capabilities over time.
All presented integrated robotic scene perception
approaches (see Table 1) build up prelearned skills except
for the method of Wyatt et al. [67], which allows modifications
of the perception after deployment (see the " The
Holistic Scope of the Latest Integrated Approaches " section).
Especially in the future, when robots will be deployed for
long periods of time, the initial perception will become
obsolete unless the perception is adjusted continuously to
the environment. Therefore, a gap concerns the learning of
perceptual capabilities during runtime. Although research
on robot learning, such as domain adaptation [109] and
continuous learning (lifelong learning, perceptual learning,
and never-ending learning) [110], reaches back almost 30
years [111], it is still rarely used in practical applications. A
common strategy is to learn from demonstration [112]. For
instance, a human could teach the robot to adjust a generic
perception to changes in the environment. The human can
approve or decline estimates of the perception system via a
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
