IEEE Robotics & Automation Magazine - June 2020 - 135

RMSE-y

RMSE-xy

OpenPose actual versus GT

39.616 (49.049)

31.014 (45.166)

38.634 (52.428)

Ours versus OpenPose actual

58.235 (33.514)

33.486 (22.751)

49.689 (24.653)

Ours versus GT

68.781 (47.798)

43.546 (40.845)

62.000 (45.836)

Ours: bb

0.251 (0.148) =ˆ 25.1 (14.8) cm

0.063 (0.055) =ˆ 12.6 (11) cm

We used the PredNet module trained from JAAD to test
on the PedX data set. Table 3 presents the pose-prediction
results on PedX. The top row shows the RMSE results
between pose detected by OpenPose on the actual video clip
and the manual GT annotation. The second row shows the
RMSE results between our proposed method and poses
detected by OpenPose on the actual video clip. The third row
shows the RMSE results between the OpenPose pose on the
predicted video clip (i.e., our proposed method) and manual
GT annotation. The RMSE-x, RMSE-y, and RMSE-xy columns correspond to the RMSE results on the x-axis (width of
the image), y-axis (height of the image), and average of both,
respectively. The lower the RMSE results, the better the prediction performance. The first three rows show the RMSE
results in pixels. The bottom row shows the percentage of
pose-prediction error normalized by bb sizes, along with their
corresponding error values in metric space in centimeters.
Since the PedX data set contains manual GT annotations,
we compared our pose-prediction results to both the manual GT and the poses detected by OpenPose on the actual
video clip. We also evaluated the accuracy of the OpenPose
keypoint detector compared with the manual GT. Compared with Table 2, the PedX prediction results have slightly
higher RMSE than when tested on the JAAD data set due to
1) the frame rate being five times higher in JAAD than PedX
and 2) the PedX scenes having never been observed in
training. Nevertheless, using the model trained on JAAD
can still make pose prediction on previously unseen PedX
contexts with a relatively small error (25.1 cm in width and
12.6 cm in height).
Figure 3 shows visual examples of pose-prediction results
for sample sequences from JAAD and PedX. The predicted
skeleton poses were overlaid with the predicted RGB images
in rows 3 and 6. Our pose-prediction system can correctly
detect and predict human poses in the scene based on the
predicted future frame from PredNet. Particularly, when a
pedestrian was partially occluded because of the motion of
the ego vehicle (the vehicle with the data-collection camera),
as shown in the final column of row 3 in Figure 3, our poseprediction process was still able to detect the correct pose and
mark that the legs are occluded (not plotted on the image).
Multiple Time-Step Prediction
This section presents the experimental results of multiple
time-step prediction (MTP) in the future. Given 10 actual
observed frames in the JAAD data set, our PredNet-based,

(MSE)

RMSE-x

0.0125

0.95

0.0100

0.9

0.0075

0.85

0.0050

0.8
5

15
20
Time Step

(SSIM)

Table 3. The evaluation of pose prediction on the PedX data set.

Figure 5. The multiple time-step frame-prediction results on the
JAAD data set. The x-axis marks the time steps (the final 20 time
steps were extrapolated from the MTP process), and the y-axes
show the frame MSE (left, red) and SSIM (right, blue).

video-generation module extrapolates the next 20 time steps, in
which the 11th frame prediction was fed back to the network as
input to generate the 12th frame prediction and so on. Then,
our pose-prediction module performed pose estimation on the
predicted frames. Since R 0l and E 0l in PredNet were initialized
to zero, the prediction at the initial time step was spatially uniform and therefore not considered in our analysis.
Figure 5 shows the multiple time-step frame-prediction
results evaluated by the metrics MSE and SSIM. Recall that
the MSE calculates the
pixel difference between
the predicted and actual
Our proposed approach
frames (the lower the better) and the SSIM is cordoes not rely on manually
related with perceptual
similarity (the larger the
defined models and
better). The second time
step produced high frame
distributions but instead
error due to the fact that
there is no motion inforperforms frame-based 2D
mation available yet in
the sequence, causing the
pose prediction.
frame reconstruction to
be blurry. In the next few
time steps, PredNet learned
the underlying dynamics in the motion sequence, and the
predicted frames better matched the input (actual) frames,
producing relatively low MSE and high SSIM results between
time steps 3 and 10. This observation is consistent with the
description in [9]. Since previous predictions were used as
JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

135

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020