IEEE Robotics & Automation Magazine - June 2020 - 132

Velodyne Lidars

RGB Camera

Figure 2. The PedX data-collection system on an autonomous
vehicle parked at an intersection in downtown Ann Arbor,
Michigan.

Table 1. The evaluation of next-frame predictions
on JAAD and PedX data sets.
MSE (# 10-3)

SSIM

PredNet on JAAD

2.621 (0.587)

0.926 (0.017)

Copy last frame on JAAD

6.602 (0.732)

0.904 (0.028)

PredNet on PedX

1.244 (0.093)

0.962 (0.003)

Copy last frame on PedX

1.759

0.955

These no longer require accurate measurements and annotations for historic skeleton sequences, which often further
requires expensive data collection instruments such as 3D
lidar, as required by prior work. (Lidar is an instrument
commonly used for autonomous vehicle perception to
measure precise ranges of objects. High-resolution lidars
can be expensive.)
In addition, the PredNet module inherently incorporates
implicit models of scene structures and movements of objects
such as buildings, streets, and pedestrians as observed from a
moving vehicle. Moreover, previous supervised pose-prediction methods often perform poorly, with imprecise measurements or occluded body parts in previous frames. However,
our method naturally alleviates the missing-data problem
since pose detection is performed per frame and does not rely
on previous skeleton-detection results.
Experimental Results: Joint Attention for
Autonomous Driving and PedX
We present the experimental results of our proposed poseprediction approach on two real-world autonomous-vehicle-perception data sets, Joint Attention for Autonomous
Driving (JAAD) and PedX. The JAAD dataset [13]-[16] contains 346 video clips collected from one of three different
high-resolution monocular cameras in North America and
Europe. The camera is positioned inside the car below the
132

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

rear-view mirror. The frame rate is 30 frames/s (FPS). PedX
[17] is a large-scale multimodal data set for pedestrians collected at intersections in downtown Ann Arbor, Michigan,
in 2017. The data set provides high-resolution stereo images
and lidar data with manual 2D ground-truth annotations.
Data were captured using two pairs of stereo cameras and
four Velodyne lidar sensors.
Figure 2 shows the data-collection configuration of the
PedX data set. In the following experiments, we use data
from only one of the forward-facing cameras with a 6-FPS
frame rate. Both JAAD and PedX provide car-mount RGB
videos in urban traffic scenes and observe pedestrian movements in the wild. The JAAD data set was collected from a
moving car (thus, moving background and moving pedestrians), while the PedX data set was collected from a parked
autonomous vehicle (stationary scene, moving pedestrians).
The JAAD data set also includes a variety of traffic scenes,
such as roads and parking lot, while PedX focuses on three
urban intersections.
The JAAD data set contains 81,906 frames in total. We
divided it into 30-frame sequences (1 s long) and used 85% of
the sequences that were randomly shuffled for training, 5%
for validation, and the remaining 10% for testing (8,280
frames). We used the same parameters of the PredNet model
described in [9] and trained the model from scratch. The
input images were downsampled to 256 × 456 to accommodate the computer memory while maintaining the aspect
ratio. We also divided the PedX data set into 30-frame
sequences (approximately 5 s long) and used the PredNet
module trained from JAAD to test all 484 PedX sequences
(14,520 frames).
Frame-Prediction Results
We first report the quantitative evaluation of the predicted
future frames as compared to the actual collected video
frames. Two metrics are used: the mean squared error (MSE)
and the Structural Similarity Index Measure (SSIM) [18]. The
MSE calculates the pixel difference between the predicted and
actual frames (the lower the better), and the SSIM is correlated
with perceptual similarity (the larger the better). We also
report the results from the trivial solution of copying the last
frame as baseline. As shown in Table 1, the PredNet module is
effective in predicting the future frame and achieves better
performance in both MSE and SSIM on both the JAAD and
PedX data sets. The MSE is lower and the SSIM higher for the
PedX data set overall since the background scene in PedX data
is fixed while the background changes as the car moves in the
JAAD data set. In the table, the best performance is in bold,
and the standard deviation across three runs is presented
in parentheses.
Figure 3 provides visual examples of the future frame-prediction results of PredNet on sample sequences from JAAD
and PedX. We observed that the frames predicted by PredNet
(rows 2 and 5) were very similar visually to those from the
actual video clips (rows 1 and 4). The frame-prediction step
was also capable of extrapolating the sky and textures of other

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020