IEEE Robotics & Automation Magazine - June 2020 - 130

guidance regarding various exhibitions and some who may
intentionally walk up to it or block its way to "test" the system.
A self-driving car or a delivery robot can regularly find itself
navigating a business district while trying to avoid a collision
in the midst of heavy pedestrian traffic. In industrial settings,
collaborative robots on assembly lines must allow humans to
stand in their proximity and learn to recognize the intentions of
human workers based on their movements and gestures to
ensure safety and improve worker ergonomics and productivity
[3], [4]. As these examples suggest, it is important for a robot or
automation system to correctly understand and predict human
poses to determine the intentions of humans and to make
appropriate decisions and
actions to avoid collisions
and facilitate human-
Our pose-detection
robot collaboration.
A human pose is comsystem generates future
monly represented using
a kinematic skeletal sysframes based solely on
tem of body joint locations, known as keypoints
a car-mounted camera
in the perception community. Given sequences
and performs direct pose
of human poses extracted
from collected data, such
estimation per frame.
as images or videos in the
past, the goal of pose prediction is to capture the
underlying motion model of humans and make inferences
about the pose and location of a person or multiple people at
future time steps. There are a number of challenges associated
with pedestrian pose estimation and prediction, including
body part occlusion by other pedestrians or vehicles, human
appearance and clothing variety, data collection range, camera
viewpoint and lighting conditions, complex traffic scenes in
the background, and stochasticity in human motion [5]-[7].
Various machine- and deep-learning (DL) methods have
been proposed for human pose prediction based on sequences of motion data, including probabilistic dynamic models,
physics-based models, supervised deep neural networks, and
methods based on video generation. DL methods, such as
recurrent neural networks (RNNs) and, in particular long
short-term memory (LSTM) networks [8], have shown superior performance compared with traditional (nondeep)
machine-learning and physics-based methods for pose prediction. However, supervised DL methods for pose prediction
in the literature typically require accurate skeleton annotations of training sequences, which may be difficult or expensive to obtain. If prior annotations are imprecise or erroneous
or if they contain missing frames, the performance of supervised DL methods may become subpar.
In this article, we developed an unsupervised method for
pedestrian pose prediction based on car-mounted monocular
camera videos collected by an autonomous vehicle. Our proposed system uses PredNet [9] as an unsupervised video-prediction method to generate future predicted frames and perform
130

IEEE ROBOTICS & AUTOMATION MAGAZINE

JUNE 2020

image-based pedestrian pose detection. This eliminates the need
for accurate measurements and prior skeleton annotations, as are
required by standard supervised pose-prediction methods.
PredNet-Based Unsupervised Pedestrian
Pose Prediction
This section describes the foundation of our future framegeneration module, PredNet, and our proposed unsupervised
pose-prediction pipeline.
PredNet
PredNet [9] is a deep neural network inspired by the concept
of predictive coding from the neuroscience literature. The
PredNet architecture consists of four basic layered units:
recurrent representation (R), prediction ( At ), the input convolutional layer (A), and error representation (E). Figure 1(b)
shows the four-layer PredNet structure used in our system.
Given a sequence of images (video data) as input, the target
value of the lowest layer A 0 is set to be equal to the image
sequence. PredNet goes through top-down and bottom-up
passes to calculate the states of all units in each layer. First, at
the top-most layer (l = 3) at time step t, the recurrent prediction state R t3 is updated by passing a copy of the error signal
at previous time step E t3- 1 and the recurrent prediction
state at previous time step R t3- 1 through the convolutional
LSTM units. For all subsequent layers l = 0, 1, 2, the recurrent
prediction state R tl is updated according to E tl - 1, R tl - 1, and an
upsampled copy of R tl + 1.
At each layer, the prediction At tl is computed by a convolution of R tl followed by rectified linear unit (ReLU) for nonlinearity. The bottom layer At t0 is additionally passed through a
saturating linear unit (SatLU) to make sure the predictions of
the next frame do not exceed the maximum pixel value. The
error response at the bottom layer E t0 is calculated by passing
the difference between At t0 and A t0 (predicted image and
actual image, respectively) through a ReLU and then dividing
it into positive and negative errors and concatenation. For l =
1, 2, 3, the errors from the layer below E tl - 1 are passed
through a convolution unit, followed by ReLU and max pooling, to become the input to the next layer A tl . PredNet can be
trained end-to-end using gradient descent by minimizing the
firing rates of the error neurons. The prediction state of the
lowest layer At t0 is returned as the prediction result for time
step t. This process is repeated for all time steps t = 1 " T in
the given image sequence. The pseudocode of the PredNet
update process can be seen in algorithm 1 in [9].
Our pose-prediction network is based on, but not limited
to, PredNet. In fact, the video-prediction module (shaded yellow in Figure 1) can be easily swapped with other video-prediction methods. We favor PredNet in our system as it has
demonstrated superior performance for future frame prediction with moving backgrounds (as demonstrated in the original article on the KITTI benchmark suite [10]), which fits our
goal of predicting pedestrian poses from data as typically
observed by an autonomous vehicle-perception system. Also,
PredNet is an unsupervised method, which has the advantage

IEEE Robotics & Automation Magazine - June 2020

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2020