IEEE Robotics & Automation Magazine - June 2020 - 134

Table 2. The evaluation of pose prediction on the JAAD data set. The equal sign means "corresponds to."
RMSE-x

RMSE-y

RMSE-xy

Copy last frame

102.534 (165.329)

89.801 (137.015)

101.561 (148.418)

Frame difference

92.456 (151.035)

78.352 (117.988)

89.816 (132.828)

LSTM-3LR

72.366 (125.301)

57.324 (95.502)

68.374 (109.531)

Ours

29.347 (39.818)

19.088 (21.685)

26.516 (30.623)

Ours: bb

0.145 (0.144) =ˆ 14.5 (14.4) cm

0.040 (0.042) =ˆ 8.0 (8.2) cm

32
30
28
26
24
22
20
18
16
14

Figure 4. The predicted skeleton RMSE results of our proposed
method in pixels. Each circle represents a keypoint (a joint) on the
body. The 25 keypoints include left and right shoulders, elbows,
wrists, hips, knees, ankles, eyes, ears, feet, toes, heels, nose, neck,
and center of hip. The colors of the circle represent the mean
RMSE results across three runs; yellow shows higher error and
dark blue shows low error. The size of the circle represents the
standard deviation value at each joint; the larger the circle, the
higher the standard deviation.

Mask R-CNN, based on the actual video frames, as the ground
truth (GT). We then compared the pose-detection results from
our proposed approach (based on the predicted future frames)
to the GT and calculated the root-mean-square-error (RMSE)
results. In PedX, we also report the RMSE results between our
pose-prediction results and the manual 2D annotations. We
compared our approach with two previous supervised DL
pose-prediction methods, LSTM-3LR [6] and a frame-difference method adapted from Bio-LSTM (Bio-LSTM-L c ) [19].
For LSTM-3LR, we used a stacked, three-layer LSTM with
64 units and 250 training epochs in our experiments. For the
frame difference method, we used the most prominent gait feature, the gait periodicity loss (L c) in the Bio-LSTM objective
function, with a two-layer, stacked LSTM RNN for pose prediction. We also reported the RMSE results from the naïve baseline by copying poses from the last frame.
134

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

Table 2 presents the pose-prediction results of the JAAD
data set. In the table, the best performance is in bold, and
the standard deviation across three runs is presented in
parentheses. The RMSE-x, RMSE-y, and RMSE-xy columns
correspond to the RMSE results on the x-axis (width of the
image), y-axis (height of the image), and average of both,
respectively. The lower the RMSE results, the better the prediction performance. The first four rows show the RMSE results
in pixels. The bottom row shows the percentage of pose-prediction error normalized by bb sizes, along with their corresponding error values in metric space in centimeters.
Our proposed approach yields smaller RMSE mean and
standard deviation values than comparison methods in both
the x and y (width and height) directions, indicating its
effectiveness and consistency. The two supervised comparison methods, LSTM-3LR and Bio-LSTM, were both originally developed for 3D skeleton/mesh prediction and
require accurate full-body annotations in previous frames.
In this experiment, where 2D key points detected in previous frames are often imprecise or missing due to occlusion,
supervised methods produced a higher error. The trivial
solution of directly copying the last frame yielded the highest error in both height and width directions, which is as
expected since this baseline method does not account for
pedestrian motion between frames at all. Our proposed
pose-prediction method, on the other hand, produces pose
estimations based on the predicted future frame and does
not require previous skeleton-detection results, which
resulted in significantly fewer errors and more consistent
pose-prediction results.
Figure 4 plots the skeleton RMSE results in pixels at each
joint. As shown, the body center has the lowest error whereas
the extremities (hands and feet/heels) show both larger mean
and standard deviation error values. This is understandable as
the hands and feet are the most flexible parts of the human
body and also the most easily occluded; they are, therefore,
the most difficult to predict. To further translate the imagelevel evaluation (in pixels) to metric space (in meters), we
computed the pose-prediction errors normalized by the
heights and widths of the corresponding bbs detected by
Mask R-CNN. The bottom row in Table 2 shows the percentage error with respect to human bb sizes. Assuming that a
person is approximately 2 m tall and 1 m wide, our average
pose error in the physical space is approximately 14.5 cm in
the lateral direction and 8 cm height-wise.

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020