IEEE Consumer Electronics Magazine - January/February 2020 - 97

In spite of the intensive work conducted on
hand gesture recognition, the proposed solutions
in the literature are almost limited. They have
shortcomings or work under some restrictions.
Some of the aforementioned methods, for example, required complex hardware setup, a specific
dressing or colored gloves as given by Kim and
Yoon.11 Others had been tested on simple static
gestures,5 very limited number of gestures5,10 or
tested only in signer dependent mode.7,8 To bridge
the gap of need and overcome the different barriers, we proposed a vision-based robust method
in this article. The proposed method does not
require any hardware setup. It neither requires a
specific dressing nor colored gloves. It was tested
for signer dependent and independent modes on
different datasets of various sizes.

Table 1. Selected gestures from the KSU-SSL dataset.

DATASETS DESCRIPTION
The method in this article was evaluated on
hand gestures from three different datasets:
namely, the King Saud university Saudi sign language (KSU-SSL) dataset, ArSL dataset collected by
the College of Engineering in the American University of Sharjah,8 and the Purdue RVL-SLLL American Sign Language (RVL-SLLL-ASL) dataset.12 The
first dataset (KSU-SSL Dataset) is still under development. The dataset contains selected gestures
for the most common words/phrases and sign language expressions in Saudi Arabia. The recorded
gestures are a mix of single-handed and twohanded gestures. More than 40 subjects are
involved to record the dataset over five sessions.
There are no constraints on the participants clothing. Different devices such as Microsoft Kinect and
RGB cameras are used for recording this dataset.
Table 1 gives 40 gestures selected from KSU-SSL to
test our system. The ArSL dataset contains 23 gestures, with 150 repetition per gesture.8 The Purdue
RVL-SLLL-ASL dataset contains 43 isolated ASL
gestures.12 The following ten gestures were
chosen to test the system: "Away," "Up," "Down,"
"Left," "Right," "Inform," "Happen," "Skilled,"
"Illegal," and "Influence."

PROPOSED SYSTEM
The proposed system in this article consists
of two main phases: 1) video preprocessing and
2) feature learning and classification.

January/February 2020

Video preprocessing: In this phase, the input
videos were converted into sequences of RGB
frames of different lengths. To apply 3D CNN on
the input sequences, we need to unify the spatial
and temporal dimensions of all sequences. For
that reason, we started by resizing each frame to
a fixed size of 112 Â 112 pixels to reduce the computation cost of the successive steps. Next, a linear mapping was used to unify the length of the
frame sequences. Whatever the length of an input
sequence, it is linearly mapped to a 16 frames
sequence. The indices of the selected frames
from the input sequence are calculated as

lenðinputÞ
indexi ¼ round
Âi
16

(1)

where len(input) is the length of the input
sequence and i ¼ 1 : 16.
Feature learning and classification: In this phase,
we employed the transfer learning to overcome
the lack of huge annotated gesture dataset. We initialize the 3D CNN with the C3D structure13 trained

IEEE Consumer Electronics Magazine - January/February 2020

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - January/February 2020