IEEE Robotics & Automation Magazine - March 2022 - 51

to transform spatial data to or from the robot's other reference
frames. For the purposes of our system, these
approaches are interchangeable. They both provide the
devices with a common coordinate system to share spatial
information, but the shared map approach is tailored to
robots equipped for lidar navigation, while the ASA
approach is designed for robots using visual navigation.
Gesture Recognition
In the colocalized scenario we consider here, egocentric sensing
by human-oriented devices can give spatial meaning to
human motions, expressions, and gestures. We focus on
detecting and classifying hand gestures with a HoloLens 2
headset and translating these into navigation commands for
the robot. The HoloLens 2 features a variety of cameras for
sensing the environment and actions of the user and estimating
how the device moves through space. In particular, we
leverage its hand tracking capability, which uses the depth
camera to track the user's articulated hands and exposes the
joint positions through an API.
We implement our gesture classification model as a neural
network. Namely, we use a multilayer perceptron (MLP) [10],
[11], which takes as input HoloLens hand tracking data and
outputs probabilities across a set of predefined gesture classes.
As for the hand data, we assume that only one hand is in view
(or we consider the right hand if both are in view) and extract
local angles for a total of 19 joints. In our experiments, we
found that using only joint flexion angle values is enough to
obtain an accurate gesture classification. We therefore parameterize
each joint by a single value. Relying on single-frame
predictions can lead to noisy classification results. To improve
robustness, we leverage temporal information, running the
model through time windows of 12 frames (which correspond
to roughly 0.2 s with the app running at 60 frames per
second): to classify a gesture at time t, we consider all the
hand joint angle values tracked within the interval [t, t − 12),
flattening them into a (12 × 19)-dimensional vector that constitutes
our MLP input. For the output, we consider three
main gesture classes for human-machine interaction: " stop, "
" come here, " and " point. " We also add a background class to
identify frames in which the user is not interacting with the
robot and therefore not performing any gesture. This gives us
a total of four classes.
The MLP network has a total of four layers and uses rectified
linear units (ReLUs) [10] as activation functions. All hidden
layers are 128-dimensional. Per-gesture confidence values
are obtained by applying the sigmoid function to the output
of the last layer. Finally, the gesture with the highest confidence
is chosen as the classifier output. Figure 5(a) exemplifies
the app output: the first line describes the output gesture
( " stop " ), which is the one obtaining the highest confidence
value (0.62). We found that adding an attention layer [12]
right before the classification one helps capture spatial and
temporal correlations among joints and therefore yields more
accurate results. The network produced good results when
trained on a small training set: we asked six subjects to twice
perform each of the three gestures (plus random actions for
the background class), recording the corresponding hand
tracking data with an ad hoc app. We trained our model on
data from five subjects, withholding one subject for validation.
We implemented our MLP network in PyTorch and
reimplemented it in C# to perform inference in Unity.
Robot Control With Holograms and Gestures
In this system, the HoloLens acts as a human-robot interface
for giving the robot navigation commands. The user can
select an arbitrary navigation goal for the robot by moving a
holographic marker to a location in the environment and
confirming this as the robot's target pose by clicking a holographic
button [see Figure 5(c)]. Alternatively, intuitive hand
gestures (see the " Gesture Recognition " section) are detected
on the HoloLens and translated to navigation commands for
the robot. Figure 6 illustrates the components of the system
and flow of data among them.
In the case of a navigation goal set by the holographic
marker, the target pose is selected in the reference frame of the
HoloLens and transformed to the common reference frame of
the map that is shared between the robot and HoloLens so that
(a)
(b)
(c)
Figure 5. Human-robot interaction through hand gestures and navigation goals. (a) The user makes the " stop " gesture, which the
classifier correctly identifies. This gesture is mapped to an action that preempts and cancels an existing trajectory on the robot. (b)
Another gesture that is recognized is " come here, " which triggers the robot to plan a path to the position of the user, which it knows
due to colocalization. (c) Finally, the robot can provide visual feedback of its intent, as seen by its planned path.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
51

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022