IEEE Consumer Electronics Magazine - July/August 2022 - 47

Figure 1. The flowchart of proposed method.
detection and tracking, ii) hand luggage localization,
and iii) abandoned object decision. The
proposed system was designed for a CCTVbased
surveillance environment, and the detail
of each part of the system will be given in three
sections, respectively.
Pedestrian Detection and Tracking
The first part of the proposed system detects
and tracks a pedestrian who potentially carries
hand luggage to recognize the context of abandoned
situation. Results of pedestrian detection
and tracking can provide the distance between
the hand luggage and the owner. We used the
RetinaNet detector11 to detect pedestrians, and
a kernelized correlation filter (KCF)-based
tracker12;13
to track them. In outdoor environments,
many challenging factors cause tracking
failure, such as changes in illumination, rapid
movement, and occlusion when tracking pedestrians.
The failure of pedestrian tracking in the
proposed abandoned object detection system
causes false detection of hand luggage, which
results in failure in detecting abandoned objects.
As shown in Figure 2, the pedestrian detection
and tracking module first estimates the
coordinate of a pedestrian, and tracks it using a
KCF tracker. The KCF tracker has the two advantages:
i) efficient computation by replacing the
correlation operation in the spatial domain with
the element-by-element operation in the Fourier
domain, and ii) robustness to shape changes
using online training. In the surveillance system,
it is necessary to check if a new pedestrian
appears or a pedestrian being tracked does not
exist in the image along with the tracking failure.
RetinaNet detector was used to detect pedestrians
with additional datasets acquired by the
on-site CCTV environment to perform training.
Because the tracker is updated through detection
at a regular interval, retracking is possible
by updating the detected pedestrian even if the
tracking fails. In the proposed tracking method,
the same pedestrian is retracked by comparing
the similarity between the pedestrian being
July/August 2022
Figure 2. Block diagram of the pedestrian detection
and tracking module, which is represented by the
first block in Figure 1.
tracked and the detected pedestrian. To compare
the similarity, we used a histogram-based
method for robust tracking against challenging
environment such as occlusion.14
HLD Network
Existing methods detected a static abandoned
object using background subtraction, which
results in false detection such as blinking signs
and moving objects. To solve that problem, the
proposed method tries to detect an abandoned
object using a deep neural network that localizes
hand luggage abandoned by a person. The proposed
network merges two networks to detect an
unspecified object located in the hand. The
merged network consists of a keypoint detection
network to detect the hand location and an
object detection network15
to detect various
objects including hand luggage. The keypoint
detection process obtains a human skeleton by
detecting the points of each joint. The obtained
skeleton plays a role in recognizing an action. In
this article, we used the OpenPose network16 for
the latter purpose. The OpenPose network computes
a confidence map to locate joints and an
affinity field to connect the joints. The human
skeleton can be obtained by connecting the confidence
map of the joint along the direction of the
affinity field. This algorithm improves runtime
performance and accuracy in the sense of part
affinity fields (PAF). In this article, this keypoint
detection and SSD object detection are merged to
detect hand luggage. The SSD network is widely
used in a surveillance system since it can detect
an object detection in real time.
In the object detection network, the feature
map is divided into multiple scales to detect
objects. There are six feature maps of sizes
3838, 1919, 1010, 55, 33, and 11.
The reason for dividing the scale is to detect
47
IEEE Consumer Electronics Magazine - July/August 2022

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - July/August 2022
