IEEE Robotics & Automation Magazine - June 2020 - 144

More details of the loop-closure detection approach can be
found in [8]. Having a pair of loop-closure candidates (two
point clouds correspond to the same place), the next step is
to calculate their relative pose, i.e., point cloud registration.
FlowNet3D [11], as the state of the art for point cloudbased scene flow estimation, is utilized in the loop-closure
module to calculate the rigid motion between point clouds.
More specifically, we first shift the target point cloud
according to the scene flow predicted by FlowNet3D and
perform singular value decomposition to estimate the rigid
motion between the target point cloud and the shifted
one. Then we conduct ICP to estimate the rigid motion
between the shifted target point cloud and source point
cloud. By multiplying these two rigid motions, we can calculate the final point cloud registration result, thus obtaining the loop-closure constraint.
For the mapping task in large-scale, dynamic environments, loop-closure detection and point cloud registration
are the most challenging elements. As shown in [8], the
state-of-the-art approach [7] may also fail in the presence
of large driving distances. The sparse static features in the
dynamic environment and the cumulative errors in the
large-scale mapping process bring great difficulties. Therefore, we introduce deep learning techniques to generate
more discriminative and generalizable place descriptors to
improve the accuracy and robustness of loop-closure detection and present a point cloud registration approach by
combining the deep learning technique and ICP method to
achieve loop closure. Unlike existing approaches [2], [7],
the proposed approach does not need an accurate initial
pose estimation (which requires accurate odometry and
robust differential GPS information and is difficult to
obtain since the cumulative error could be very large in the
long-distance mapping process).
Based on the previous three modules, the traditional frontend pose tracking and back-end pose graph optimization
framework can be directly implemented to build a complete
SLAM system:
● Pose tracking: In the front end, we directly utilize the stateof-the-art LeGO-LOAM [7] for pose tracking. The 3D

Laser Point
Cloud

Laser Detection

LDR

laser point cloud is preprocessed to extract handcrafted
features from the ground, edge, and planar points, and the
ego-motion pose is estimated and updated iteratively by
combining the odometry-based pose-prediction and feature-matching result.
● Point cloud mapping: In the back end, we utilize the factor
graph model to estimate the pose graph that is constructed
by a series of continuous key frames. The key frames are
constructed considering the driving distance, steering
angle, and time duration. Then the loop-closure constraints obtained from the loop-closure module are
imported to optimize the pose graph and prevent the mapping error from quickly accumulating over long distance
pose tracking.
For the real-time localization task, we use the similar posetracking and key frame construction methods. Through associating the constructed key frames with those in the prior
map, the vehicle pose can be estimated in real time.
Multisensor Fusion Based Perception
The perception system needs to achieve the object detection,
tracking, and motion estimation tasks. Currently, widely used
approaches include visual- and laser-based ones. Visual sensor
data can provide rich information and abundant features but
suffers from a limited perception range and is sensitive to illumination changes. The laser-based perception approach can
provide accurate spatial information of the detected object
and is more robust at night or in bad weather, but it has local
sparsity problems; in addition, it is difficult to obtain complete and accurate object segmentations. Considering their
strong complementarity, as shown in Figure 5, we propose an
integrated perception framework based on multisensor information fusions.
● Joint calibration: To carry out the integrated perception
framework, joint calibration needs first to be conducted
between the laser and camera. The calibration accuracy
limits errors occurring in the integrated perception results.
Most existing methods [12], [13] utilize the special calibration board and its spatial information to calculate the projection matrix. However, directly implementing these

OMER
Laser Tracking

LTR

LDR
Joint
Calibration

Fusion FDR
Detection

FDR

LTR

Tracking
Template Updation

VTR

VDR
Image

Visual Detection

VDR

Data Fusion and Object Detection

OMER

Visual Tracking

Vehicle Environment
State
Map
Object Motion OMER
Estimation

Object Motion Estimation
VTR

Object Tracking

Figure 5. The overall structure of the proposed multisensor information-fusion-based perception system. LDR: laser detection result;
FDR: fusion detection result; VDR: visual detection result; LTR: laser-tracking result; VTR: visual-tracking result; OMER: object motion
estimation result.

144

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020