IEEE Robotics & Automation Magazine - December 2022 - 41

their tasks proactively. For instance, if the person is cooking
in a room, it is very likely that this room is a kitchen in which
the robot can act accordingly [83].
Previous work [84] focuses on learning human actions by
observing humans in their daily life. They merge joint
motions and locations concerning the landmark points on
the map. Kostavelis et al. [85] propose using object recognition
and skeleton-based action recognition to make their
social robot understand human actions. They deploy the
robot in real home environments to test their system. The
previous work [86] employs a long short-term memory [87]
network on the robot to greet the user. The background may
be misleading for the algorithms that are using appearance
features. That is why generating action proposals may
improve performance, and this is essential for mobile robots
because the background in the robot view is dynamic [88]. In
contrast to the spatial-temporal interpretations of a single
instance, there are a few approaches that interpret relations
within multiple instances. For example, Philipp et al. [89]
propose to use Bayesian networks to estimate on which
object the user's attention focuses.
The attention-sensitive functionalities as high-level scene
interpretations are essential for improving human-robot
interaction. More complex interpretations go into the affordance
estimation [90]. Herewith, robots know what can be
done with objects based on past observations, probabilities,
and personal data (such as emotions, preferences, and relations).
For instance, knowing that a pod can be used for cooking
offers novel capabilities for robots. Affordances link
perception into the cognitive capabilities that are fundamental
for interpreting scenes.
Implications
Recent research on robot perception displays similarities to
fundamental theories on human perception. Figure 3 visualizes
the transferred scene perception process. Humans and
robots recognize information from the scene by making sensor
data understandable. The human brain and the robot
storage, respectively, represent the knowledge in multiple layers
in a known structure. Compared to humans, robots are
capable of using external perceived data within their original
format, whereas humans have to exchange perceived data
through verbal, visual, or written communication. The interpretation
of knowledge over multiple dimensions, such as
time, space, and relations, enables a high-level scene understanding
that improves the cognitive intelligence of robots.
Herewith, humans highly benefit from life-long learning,
whereby robots can benefit from shared and initial data.
In recent years, robotic recognition and interpretation use
deep learning, such as CNNs and generative adversarial networks
[91], based on artificial NNs (ANNs), to solve a very
specific task [92] as described previously. Inspired by biological
NNs [93], ANNs consist of multiple layers of connected
artificial neurons [94]. The weight assigned to these neurons
relies on an initial ANN training using a high amount of
labeled data. This in turn enables the ANN to compute labels
of unknown data via inference. In particular, deep learningbased
approaches, such as ViTs, gained popularity in robotics
due to their superior performance on continuous data
streams. ViTs, coming from NLP, split images into fixed-size
patches [56]. Machine learning, such as deep learning,
achieves outstanding
results for many recognition
tasks [95], [96].
However, the division of
preattentive and postattentive
feature processing
can neither be strictly
adhered to nor easily ex -
tended. The inseparability
of ANNs challenges
their flexibility and reusability
due to the " black
box " characteristic. Additionally,
they require enhanced tensor computation power
that needs special attention when pushing mobile robots to
the real world [92].
Recent research on robot
perception displays
similarities to fundamental
theories on human
perception.
The Status Quo of Holistic Scene Perception
The transfer reveals similarities and differences in the perception
process between humans and robots. But how much
does the status quo of robotic scene perception cover holistic
capabilities? To answer this question, we first deal with the
challenges of fundamental technologies to extract possible
boundaries for human-like perception. Afterward, we analyze
the scope of most holistic robotic approaches for everyday
environments to answer how much they already cover the
holistic scope.
From Narrow Toward Holistic Scene Perception
Research on robot scene perception focuses on sensor-close
processing by affordable sensors. The availability of cheap
Recognition of Information
by Making Sensor Data
Understandable
Interpreting Knowledge
by Space, Time,
and Relations
Representing Knowledge
in Multiple Layers
External Data Exchange
(Robots Only)
Figure 3. A summary of the three steps of the transferred scene
perception.
DECEMBER 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
41
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
