IEEE Robotics & Automation Magazine - June 2020 - 137

alternative sensor modalities to compensate for the lighting
and other environmental effects on RGB cameras.
Analysis of Computation Time
Our proposed network was implemented in Python 3.6
using the Keras framework [20]. The PredNet module was
trained on a desktop computer with an Intel Xeon 2.10-GHz
CPU with four NVIDIA TITAN X graphics processing
units and 128-GB memory. The memory requirement can
be reduced if the input video is of lower resolution or if the
sequence length is smaller. After PredNet was trained, the
pose detection module was performed on a desktop computer with an Intel i7 3.60-GHz CPU with two NVIDIA
TITAN X GPUs.
Figure 8 shows a breakdown of our end-to-end computation time for the JAAD data set experiments (nextframe prediction). PredNet training and upsampling are
the two most time-consuming tasks. However, the PredNet training step can be omitted when a pretrained PredNet model is applied to other data sets, such as in our
PedX experiments. The training time also varies if the
input data size changes. The inference time for the PredNet testing stage is approximately 20 ms per frame. The
OpenPose pose-detection step is also quite fast, taking
approximately 418 ms/frame. In our current implementation, since both our pose detector and noise filter are
image based, we saved all upsampled RGB images to disk.
Therefore, the upsampling and Mask R-CNN filter steps
include the long disk read and write time. Future work
will include investigating non-image-based pose-prediction methods. Additionally, the pose-detection and filtering steps can be trivially parallelized.
Future of Pose Prediction and Conclusions
This article presented an unsupervised pedestrian pose-prediction system based on PredNet for autonomous vehicle perception. Our system combines video-generation approaches,
such as PredNet, and frame-based pose detectors, such as
OpenPose and Mask R-CNN, and shows effective future
pose-detection performance on real-world autonomous vehicle applications.
There are several new research directions related to this
research. This article focused on autonomous driving-perception applications, but human pose prediction can be widely
applied in various robotics and automation applications,
including virtual reality, sports and artist posture analysis,
and medical assistance. In many robotics and automation

(a)

(b)
Figure 7. Examples of OpenPose false alarms, where (a)
a windshield wiper and a person on the billboard were
misidentified as pedestrians and (b) tree branches were
misidentified as pedestrians due to dark shadows under glare.

applications with abundant perception data, accurate annotations are expensive and difficult to obtain. Therefore, it is necessary to further develop unsupervised and semisupervised
learning methods given sparse and imprecise labels for pedestrian pose prediction in various contexts. Particularly, with
human pose analysis and sequence prediction, additional
investigations can be conducted to incorporate more realistic
spatial, temporal, textural, semantic, and biology-derived constraints in the learning model [21].
Furthermore, mobile robots and automation systems are
interacting with the real world. In autonomous driving, pedestrians and vehicles are constantly making decisions based on
their interactions and finding the balance between achieving
certain goals and avoiding risk and collision. Interesting future
work will include incorporating pedestrian-pedestrian and
pedestrian-vehicle interactions in pose and trajectory prediction
and using such prediction results for activity inference, low-level
decision making (such as stop/go), and path planning.

Runtime

Downsample: 2 h 42 min
OP Pose Detection: 58 min

PredNet-Train: 36 h 4 min
Mask R-CNN Filter: 2 h 9 min

PredNet-Test: 22 min
Compute RMSE: 3 min

Upsample: 27 h 56 min

Figure 8. Analysis of the computation time for the JAAD data set experiments, trained and validated on 73,626 frames and tested on
8,280 frames. OP: OpenPose.
JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

137

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020