IEEE Robotics & Automation Magazine - December 2022 - 43

The SLAM++ project of Salas-Moreno et al. [104] is an
early approach from 2013 that concentrates on semantic
mapping. It consists of an object-based SLAM that uses
object recognition trained on a database of scanned object
models. It is capable of detecting changes in the environment,
such as moving objects. Fusion++ [100] set its focus
on semantic mapping. It is similar to SLAM++ but runs a
mask region-based CNN (R-CNN) object segmentation to
initialize a truncated signed distance field (TSDF) reconstruction
for each object. Rosinol et al. [105] recently published
Kimera, a multilayer spatial scene perception
framework aiming to close the gap between human and
robot scene perception. Kimera uses a metric-semantic
SLAM to perform a full mesh reconstruction by TSDF volumes
with its semantic on top of localization. It recognizes
building structures as well as objects from a CAD model
match. In addition, human detection and pose estimation
extend the dynamic scene information [106]. The recognition
techniques feed its scene information into the
knowledge base, where they are represented and connected
in different ways.
Fusion++ and SLAM++, which concentrate on the semantic
mapping, are not providing details of their knowledge representation.
In contrast, the approach of Wyatt et al. [67]
focuses on the representation of knowledge gaps and uncertainties.
A layered structure of proxies, unions, and beliefs
represents the spatial scene inside a relational database. They
validate by experiments in a lab that a human tutor is capable
of helping a robot fill a knowledge gap through verbal conversation.
The robot asks for missing visual features, such as the
color and shape, to prove an object is believed to close a
knowledge gap.
Kimera, which extensively covers the recognition, is also
concentrating on spatial knowledge representation by multiple
hierarchical layers, separated by the semantic. The spatial
layers comprise the metric-semantic mesh, objects, structures,
rooms, and buildings. Dynamic scene graphs simultaneously
update scene information by linking the spatial scene
Table 1. An overview of integrative robotic scene perception approaches.
Recognition of Information
Sensory
Input
Wyatt
et al. [67]
3D
camera
Reconstruction
Metric
Static
Instances
Objects
from
user input
Dynamic
Instances Database
N/A
Knowledge
Structure
Relational Multilayer spatial
representation
by proxies,
unions, and
beliefs
SLAM++
(Salas et al.
[104])
Suchan
et al. [103]
3D
camera
3D
camera
Metric-
semantic
Metric-
semantic
N/A
Objects
by scan
model
match
Walls by
planes
Human
pose
detection
Fusion++
(McCormac
et al. [100])
SOMA
(Kunze
et al. [63])
KnowRob
(Beetz
et al. [65],
Beßler
et al. [90])
Kimera
(Rosinol
et al. [105],
[106])
3D
camera
3D
camera,
2D laser
scanner
NS
NS
NS
Metric-
semantic
Metric
Objects by
R-CNN
N/A
N/A
Objects
and
people by
CNN
NS
NS
Document
based
NS
Single-layer
spatial object
graph
NS
Spatial-temporal
representation
of entities
Single-layer
spatial object
graph
Observation,
semantic, and
interpretation
layer
Relational Ontology
graph, multilevel
of metric-
semantic, logic,
and episodic
memories
3D
camera,
IMU
Metric-
semantic
Building
structures
and objects
by CAD
model
match
N/A: not available; NS: not specified.
People
by pose
detection
NS
Hierarchical
graph
connects
spatial layers
Knowledge
Knowledge Representation
Scene Representation
Point
map Place
classification
Interpretation
Spatial-
Temporal Reasoning
Belief
verification
by human
Metric-
semantic
mesh
Metric
map and
semantic by
primitives
Metric-
semantic
mesh
Point map,
objects by
pose and
bounding box
Mesh and
poses
Episodic
memory
for
reasoning
Metric-
semantic
mesh
Place
and room
classification
Hypotheses
verification,
inner
world,
and
motion
control
N/A
N/A
N/A
Human
activities,
spatial relation,
and
pattern
N/A
Human
activities
Humancentered
common
sense
N/A
N/A
DECEMBER
2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
43
IEEE Robotics & Automation Magazine - December 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2022
