IEEE Robotics & Automation Magazine - December 2019 - 65

multiple approaches to solve a specific task. Once a satisfying solution is found, the task can be executed with
the real robot [21].
Operator Relief Through Autonomous
Control Functions
While intuitive teleoperation is superior for solving complex, unknown tasks, disaster-response missions also
include several tasks that can be automated to reduce the
high cognitive load on the operators. Such tasks might
include navigating to a desired location or grasping a tool.
Moreover, in comparison to direct teleoperation, autonomy is often faster and depends less on a reliable data link.
However, the development of autonomous skills is challenging due to the level of variability of the environment
and the tasks.
Autonomous Hybrid Driving-Stepping Locomotion
The many DoF that have to be controlled in combination
with the required precision and balance assessment put a high
cognitive load on the operator and might result in motion too
slow to be practical. We developed a locomotion planner that
combines omnidirectional driving and stepping capabilities in
a single graph-search-based planning problem [22]. The environment is represented with 2D height maps computed from
laser scanner measurements. These maps are processed to
estimate costs for the robot base and individual feet, enabling
precise planning.
Two-dimensional height maps are inadequate for representing terrains, such as expanses of fine gravel. Therefore,
the planning pipeline is enriched with terrain classification
providing additional semantics (Figure 5). A geometry-based

RGB Images

analysis derives the terrain slope and roughness from point
clouds. In parallel, a vision-based terrain classification is performed on RGB images by employing a convolutional
encoder-decoder neural network. Initial training on the
CityScapes data set [23] provides the network with reliable
representations of drivable surfaces, walls, vegetation, and
terrain. Subsequent training on custom data sets from forests
and buildings refines the terrain representations. We focused
on the classification of staircases, which have many similarities to obstacles but differ in traversability. Finally, all features
are fused to pixelwise traversability assessments (safe/risky/
obstacle/stair), which are incorporated in the planner as an
additional cost term.
The fine planning resolution (2.5 cm) and high-dimensional (7 DoF) robot representation result in rapidly growing state spaces that exceed feasible computation times and
available memory for larger planning problems. We
improved the planner to generate an additional coarse, lowdimensional (3 DoF), and semantically enriched abstract
planning representation [24]. The cost function of this representation is learned by a convolutional neural network
trained on artificial short planning tasks but generalizes
well to real-world scenes. An informed heuristic (containing knowledge about the environment and the robot)
employs this abstract representation and exploits it to effectively guide the planner toward the goal. This accelerates
planning by multiple orders of magnitude with comparable
result quality.
Autonomous Dual-Arm Manipulation
While object grasping could be executed through the telepresence suit, autonomous grasping is promising to relieve

Classified Images

Terrain Class Maps

Stair Detection

Appearance-Based
Feature Extraction

Random
Forest
Classifier
Safe Risky Obstacle Stair

Geometry-Based
Feature Extraction

Registered Point Clouds

Height Maps

Cost Maps

Figure 5. Photos and a chart showing how an autonomous locomotion cost map is generated. Point clouds are processed to height
maps to finally generate cost maps. This pipeline is enriched by parallel terrain class computation. Geometry- and appearance-based
features are extracted from point clouds and RGB images. Stairs are detected in an additional module. Outputs are merged in a
random forest classifier. Terrain classes are added to the cost computation. Note that the patch of gravel in front of the robot (red
arrow) is not recognized well in the height map. The terrain class map clearly represents this area and enriches the cost map with this
information. The cost map also shows two generated paths to a goal pose on top of the staircase (green arrow). The operator can
choose one of these paths. The red path represents a low weight for terrain class-based costs: the robot traverses the patch of gravel.
The blue path incorporates terrain class-based costs with a higher weight: the robot avoids the gravel.

DECEMBER 2019

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - December 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2019