IEEE Robotics & Automation Magazine - June 2020 - 136

and uncertainty, thus causing inaccurate and missed pose-detection results (such as the person on the right-hand side) in its
extrapolations after the first 10 time steps.

(RMSE)

40
35
30
25
20
5

15
20
Time Step

MTP: x
MTP: y
MTP: xy

Next Frame: x
Next Frame: y
Next Frame: xy

Figure 6. The multiple time-step pose-prediction results on the
JAAD data set. The x-axis marks the time steps (the last 20 time
steps were extrapolated from the MTP process), and the y-axis
marks the OpenPose skeleton joint RMSE in pixels. The solid lines
show the MTP results, and the dashed lines show the next-frame
prediction results. The red, blue, and green colors represent the
RMSE results on the x-axis (width of the image), y-axis (height of
the image), and average of both, respectively.

input to extrapolate future frames, after time step 10 (in the
MTP process), the frame errors were higher than next-frameprediction results, and the MSE increased (and SSIM
decreased) over time.
Figure 6 shows the pose-prediction results for time steps
2-30 based on the MTP frame predictions, with comparison to
the next-frame prediction results. In time steps 2-10, where the
actual frames were used as
input, the MTP process is
the same as the next-frame
The PredNet training step
prediction and they yield
comparable RMSE results.
can be omitted when a
In time steps 11-30, where
the previous prediction
pretrained PredNet model
results were recursively
iterated as inputs to generis applied to other data
ate future frames, the MTP
error is higher than nextsets, such as in our PedX
frame prediction and
increased significantly over
experiments.
time as the noise overcame
the system. Qualitatively,
the bottom three rows in
Figure 3 show the MTP frame- and pose-prediction results of a
sample sequence from the JAAD data set. As time increases, the
MTP model was still able to capture certain motions in the
scene, such as the movements of clouds and surrounding cars in
the parking lot. However, the frame predictions eventually
became more and more blurry due to the accumulation of noise
136

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

Discussion
The quantitative and qualitative results presented previously
show that our proposed PredNet-based unsupervised poseprediction approach can produce accurate results. Using
JAAD and PedX, two real-world data sets in autonomous driving contexts, we show that our proposed approach is applicable to a variety of driving scenes and the frame-prediction
models can be generalized to previously unseen environments.
Our PredNet-based frame-prediction step produces realistic future frames, accounting for both vehicle and pedestrian
motions. We also observed that parts of the predicted frames
can be blurry later in the sequences (when t increases) due to
downsampling, particularly in small-textured regions, such as
the pedestrian's facial features, or unfamiliar regions, such as
when the red car entered the scene in the middle of the
sequence in Figure 3. In PedX, the blur effect seemed more
prominent due to the fact that we used a PredNet model pretrained on JAAD and the PedX environments were not
learned in training; moreover, the spacing between frames
was longer due to a low frame rate. However, such a blur
effect did not have a significant impact on the pose-prediction
step, and we observed that our pose-prediction module can
still successfully detect the skeleton joint locations of pedestrians based on predicted future frames.
Our proposed approach does not rely on manually
defined models and distributions but instead performs
frame-based 2D pose prediction. Our approach transforms
the standard supervised learning problem, which requires
full-body pose annotations in prior sequences, into an
image-data-driven, unsupervised framework. Our frameprediction step realistically predicts scene dynamics as well
as the relative motion between pedestrians (or moving
objects) and background, and our pose-detection step naturally handles occlusion based on the OpenPose detector.
Moreover, our approach produces both the RGB future
frame as well as the future pose, which makes the pose visually interpretable within the frame.
One of the challenges associated with such frame-based
pose-prediction methods is that the performance of the
pose prediction depends significantly on the accuracy of the
human detector, i.e., the OpenPose and Mask R-CNN algorithms used in our system. Although highly effective most
of the time, human detectors can occasionally produce false
alarms, such as mistaking a windshield wiper or a billboard
painting as a pedestrian or misidentifying tree branches or
dark shadows as a person, especially under difficult lighting
conditions, such as sun glare (Figure 7). It is also challenging
at times to accurately identify multiple people moving in a
crowd at a distance. This can be solved by better positioning
the camera, such as outside the windshield wiper or on top
of the car to avoid potential interference of the wipers, using
more vigorous noise-filtering approaches, and exploring

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020