IEEE Robotics & Automation Magazine - December 2022 - 40

a human-understandable ontology that conceptualizes multiple
entities within a domain and their relationships [66].
The scene representation comprises various perceptionrelated
information, such as extracted spatial features for
navigation, manipulation, and semantically enhanced maps
[67]. The requirements for robotic knowledge representation
are high. Ideally, it must be real time capable; generic;
scalable; flexible in structure; and able to update and extend
during the robot's lifetime as well as easy to connect for
access and data sharing. Generally, there are two categories
of databases: graph based and document based. Both seem
suitable for this task as they provide comprehensive features
to cover these requirements [68], [69]. Graph-based databases,
also known as relational database management systems,
represent knowledge through relations.
Herewith, it is necessary to explicitly define relations
through a common format to link semantics through an
ontology [70]. In contrast to relational databases, documentbased
databases represent data in JavaScript Object Notation-like
documents without the need for relations or
predefined structures. Nevertheless, these databases provide
features for querying or indexing the data to model dependencies
and relations implicitly dynamically. For instance,
Kunze et al. [63] propose spatial and temporal indexing for
query relations within their document-based database.
Besides, linking the knowledge base for decision making [71]
and providing the represented knowledge to robot actions,
such as manipulation or moving, enables reactive behaviors.
The comparison of scene knowledge representation indicates
that robots have advantages compared to humans.
First, the artificial scene knowledge representation has no
memory loss due to an almost unlimited storage capacity.
Second, robots can easily share and make use of foreign perception,
while humans have to transfer knowledge into an
appropriate modality, such as verbal communication. Additionally,
humans are limited in the range of information
exchange without technical assistance. In contrast, robots
can share data in their original format over networks. The
sharing of perceptional information enables robots to
directly exchange data with the infrastructure or other
robots. Another difference between human and robot
knowledge representation is the capability of robots to start
with a preinstalled environment perception model. Robots
can use the prior information of a building information
model [72] or a partial or fully premapped environment
[64]. Using prior scene knowledge reduces the setup time,
especially when using multiple agents.
Knowledge Interpretation
Based on the available scene knowledge, this perception step
interprets existing knowledge to make sense using cognitive
capabilities. It has been proven that humans interpret the
scene; however, it is unclear how this is executed within the
brain. The research by Isik et al. [73] found that the human
brain starts recognizing view-invariant observations, such as
human actions, quickly, in around 200 ms. This suggests that
40 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2022
the brain uses the form, as well as the motions, to represent
states. Furthermore, previous work [74] proposes that the
human brain benefits from causal relations, such as temporal
continuity and spatial relations, among objects, humans, and
their actions. The prerequisite is that a known structure represents
the knowledge (see the " Knowledge Representation "
section). A high-level scene analysis encodes spatial and temporal
relations between instances [75].
For most examples, the interpretation of scene knowledge
is a trivial task for humans due to lifelong learning. The interpretation
of perceived information has been trained with preknowledge,
dependent on the culture [76, Ch. 14]; context;
and situation itself [77]. Therefore, all people have a unique
perception system based on and enriched by their environment.
Thus, human interpretation is neither predictable nor
always the same. The famous duck-rabbit illusion [78] indicates
that even the season could differ the interpretation of
the scene. Therefore, perception is influenced by environmental
factors [77].
For robots, the interpretation of the scene is still an
unsolved problem [79]. The challenge is to generate and make
use of high-level semantic knowledge to reason about the
present scene. There is no commonsense model that could be
applied to every environment without the adaptation of the
interpretations. The association of scene knowledge across
multiple dimensions, such as time and space, with or without
relational dependencies between single pieces of information,
leads to this new knowledge that improves scene understanding.
For instance, a spatial-temporal scene analysis could
reveal daily habits, such as when, how, and how often we go to
the kitchen to fetch a coffee. This example shows that the kind
of required high-level information is environment specific as
well as use case specific. The goal is to understand what is
involved and what to do when and with which objects. The
use of high-level semantics within the scene is fundamental
for complex robot behavior tasks. Herewith, we identify two
types of interpretations.
On the one hand, there is research aimed at reconstructing
and interpreting the structured part of the scene, such as
room segmentation, junction detection [32], or occlusion
reasoning for simple shapes [80]. On the other hand, there is
research focusing on the unstructured part that goes deeper
into handling dynamics. The approach presented in [81] is
one of the rare examples in which already collected scene
information is used to gather new information. Observations
of objects are anchored in the scene model, which provides
basic tracking functionality. In combination with
knowledge about the whole scene, including other objects
and their spatial and semantic relations, this is used for reasoning
about the state of occluded objects, which improves
tracking and hence the whole scene state. The tracking of
instances over multiple observations enables further interpretations,
such as detecting their action [82]. The actions of
the people together with environmental semantics are valuable
input for a robot since they usually share an environment.
So, they need to understand the actions and fulfill
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
