IEEE Robotics & Automation Magazine - June 2020 - 131

of not requiring any annotation on the skeleton or mesh of
people in the scene at prior sequences. In the following experiments, we used a four-layer PredNet structure with 3 × 3
convolutions and layer channel sizes of (3,48,96,192), the
same parameter settings as described in [9] that produced
effective results on natural image-sequence prediction.
Pose Prediction
Figure 1 provides an illustration of the full system for our proposed PredNet-based unsupervised pedestrian pose-prediction
network. Given a sequence of red, green, and blue (RGB) images from a camera video as input, PredNet was used to predict
pixel-level RGB values for the next frame. Then, based on predicted future frames, poses were extracted using pretrained
human pose detectors. In our experiments, we used the OpenPose human keypoint detector [11] to identify 25 2D key
points of pedestrians based on predicted future frames. OpenPose is available for image-based pose detection; see [22].
OpenPose is a deep convolutional neural network
(CNN)-based human pose and keypoint detector that has
shown state-of-the-art performance in real-time, skeletonbased pose detection based on videos and images containing multiple persons in various actions, including common

t=0

t=1

Output

t=2
"

Input
Video

pedestrian activities such as standing and walking. To further filter out false alarms caused by nonhuman objects,
such as tree branches, utility poles, and billboards, another
layer of human detection is applied to the OpenPose
detection results. In our system, we applied a pretrained
Mask R-CNN method [12] to estimate bounding box (bb)
locations for humans and, thus, classify humans with nonhuman objects. Pre-trained weights on the Common
Objects in Context (COCO) data set for Mask R-CNN
human detection are available in [23]. The poses detected
within the human bb are returned as the output of this system, i.e., the predicted skeleton pose for the generated
future frame.
Our system offers a solution to unsupervised pose prediction inspired by human visual learning. As the car is in
motion, human drivers are able to easily identify pedestrians and moving objects in the environment based on past
observations. Additionally, human drivers do not necessarily require the exact metric of pedestrians in past sequences
to know their pose (for example, walking in a direction that
could intersect with the car). Our pose-detection system
generates future frames based solely on a car-mounted
camera and performs direct pose estimation per frame.

+/-

t=2

Pose Detection
on Predicted
Future Frame

Pretrained Human
Pose Detector
e.g., OpenPose

Noise
Filter

Pretrained Human
Bounding Box
Detector
e.g., Mask R-CNN

Predicted
Skeleton
Pose

+/-

Predicted
Future
Frame

PredNet

+/-

A0
Pool

conv LSTM
conv
Upsample

(a)

Input

ReLU
SatLU

(b)

Figure 1. The system design for the proposed PredNet-based unsupervised pedestrian pose-prediction network. (a) The flowchart
of the proposed system. For this illustration, the input video has three time steps (t = 0, 1, 2), but the input video can have more
than three frames. (b) Details of the four-layer PredNet module where the RGB blocks represent the recurrent representation (R),
prediction ( At ) and input convolutional layer (A), and error representation (E). conv: convolutional; ReLU: rectified linear unit.

JUNE 2020

IEEE ROBOTICS & AUTOMATION MAGAZINE

131

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020