IEEE Circuits and Systems Magazine - Q2 2021 - 64

Sensors
FPGA Platform
Sensor
Stereo Camera
IMU
GPS
Synchronizer
Localization
Accelerator
Sensor Interf.
Image Signal
Processor
CPU
DMA
Mem.
Controller
DRAM
DRAM
GPU
On-Vehicle PC
Multicore CPUs
Figure 4. The computing system in our autonomous vehicle.
now wildly adopted to unify various measurements in a
global timing domain. Software-based synchronization
associates samples with timestamps at the application
or the driver layer. This approach is inaccurate due to
the software processing before the timestamp stage.
The software processing introduces variable latency
that is non-deterministic.
To obtain more precise synchronization, we uses a
hardware synchronizer implemented by FPGA fabrics.
The hardware synchronizer triggers the camera sensors
and the IMU using a common timer initialized by
the satellite atomic time provided by the GPS device. It
records the triggering time of each sensor sample, and
then pack the timestamp with the corresponding sensor
data. In terms of costs, the synchronizer is extremely
lightweight in design with only 1,443 LUTs and 1,587 registers
and consumes 5mW of power.
C. Perception on FPGA
For our autonomous vehicles, the perception tasks
includes scene understanding (depth estimation and
objection detection) and localization, which are independent.
The slower one dictates the overall perception
latency.
We evaluate our perception algorithms on the CPU,
GPU and Zynq FPGA platform. Fig. 5 compares the la103
12,892
102
CPU
GPU
FPGA
tency
of each perception tasks on the FPGA platform
with the GPU. Due to the available resources, the
FPGA platform is faster than the GPU only for localization,
which is more lightweight than other tasks. We
offload localization to the FPGA while leaving other
perception task on the GPU. This partitioning frees
more GPU resources for depth estimation and object
detection, which is benefit for reducing the perception
pipeline's latency.
As with classic SLAM algorithms, our localization
algorithm consists of a front-end and a back-end. The
front-end uses the ORB features and descriptors for detecting
and tracking key points [120], [187]. The backend
uses Levenberg-Marquardt's (LM) algorithm, a nonlinear
optimization algorithm, to optimize the position
of 3 D key points and the pose of the camera [156], [188].
The ORB feature extraction/matching and the LM
optimizer are the most time-consuming parts of our
SLAM algorithm, which take up nearly all the execution
time. We accelerate ORB feature extraction/matching
and the non-linear optimizer on FPGA fabrics. The rest
lightweight parts are implemented on the ARM core of
the Zyqn platform. We use independent hardware for
each camera to extract features and compute descriptors.
Hamming distance and Sum of Absoluated Difference
(SAD) matching are implemented to obtain stable
matching results. Compared with the CPU implementation,
our FPGA implementation achieves a 2.2× speedup
and 44 fps, and saves 83% energy.
We use LM algorithm to optimize features and posDepth
Detection
Localization
Figure 5. Performance comparison of different platforms running
three perception tasks.
64
IEEE CIRCUITS AND SYSTEMS MAGAZINE
es over a fixe-size sliding window. To solve the non-linear
optimization problem, the LM algorithm iteratively
use Jacobbian to linearize the problem and solve the
linear equation at each iteration. Schur elimination is
used to reduce the dimension of the linear equation,
thus reduce the complexity of solving the equation.
Cholesky factorization is employed to solve the linear
equation. For sliding-window based vSLAM, the Jacobian
SECOND QUARTER 2021
Latency (ms)
IEEE Circuits and Systems Magazine - Q2 2021

Table of Contents for the Digital Edition of IEEE Circuits and Systems Magazine - Q2 2021
