IEEE Robotics & Automation Magazine - June 2020 - 145

Score = a × Dist. + b × PointNum

KT =

u
s v =P
1

x
y
z
1
fx 0
0 fy
0 0

Projection Model

x
Solve T by
= KT yz
EPnP
1
cx
r11 r12 r13 t1
cy
r21 r22 r23 t2
r31 r32 r33 t3
1

Fusion Detection

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

Figure 6. The system structure of the joint calibration and fusion detection modules. Dist.: distance; PointNum: number of points.

●

Joint Calibration

●

methods in large-scale scenarios will lead to large calibration errors since the practical environment size is much
larger than that of the calibration board. To obtain an accurate result, the spatial information in every part of the
large-scale environment should be considered. In addition,
the data pairs in traditional methods are usually not strict
one-to-one correspondences. To overcome these shortages,
we present a new method based on manually selected
matching 3D/2D data pairs that dispersedly locate in the
laser frame/image frame. Instead of using the corner points
on the calibration board, the manually selected points can
cover the whole calibration environment and thus ensure
accuracy in large-scale application scenarios. Finally, we
introduce the Efficient Perspective-n-Point (EPnP) method
[14] to obtain the accurate 3D-2D projection matrix. More
details are shown in Figure 6.
Object detection: The objective of this module is to determine the human workers, vehicles, pallets of goods, and
other dynamic entities moving around.
* For the laser detection module, we first remove the point
cloud belonging to the ground surface and then cluster
the point cloud into segments, thus obtaining the laser
detection results. The ground removal operation could
reduce the data count and computation cost and make
the object segment prominent without much noise.
* For the visual detection module, we directly utilize the
state-of-the-art YOLO-V3 [15] since it satisfactorily
balances real-time performance and accuracy. We train
the network on the BDD self-driving data set [16] that
contains vehicles, pedestrians, and traffic signs under
varied weather and illumination conditions. We also
refine the network on our self-collected and labeled
industrial data set.
* With calibration results, the data mapping from 3D point
cloud to image can be developed using the frustum projection model [17]; each frustum can be projected to a
corresponding rectangle in the image plane. For the
objects that can be detected in both the point cloud and
the image, we present an integrated approach that benefits from both the object recognition ability of visual
detection and the accurate measurement ability of laser
detection. We first project the point cloud into the image
plane and find out the laser points corresponding to the
object area in the image. Then, we cluster those laser
points to generate segments, remove foreground occluders and background clutter, and extract the object by the
depth information. What's more, as shown in Figure 6, a
scoring operation is used to evaluate the saliency of the
objects (to define the priority) by considering segment
locations and densities. More details of the scoring operation can be found in [18].
Object tracking: The industrial environment is usually
highly dynamic and narrow. Autonomous vehicles need to
share the operation environment with a large number of
moving vehicles and man-made objects. Since missing
detection and false detection cannot be avoided in practical

145

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020