IEEE Robotics & Automation Magazine - March 2022 - 23

inputs and movements to execute tasks, they rarely consciously
consider individual movements. Instead, subconscious
processes abstract away low-level details of control,
facilitating higher-level reasoning to ease decision making
[7]. In this work, we explore how planning could help affordance-aware
AR HMD UIs overcome this limitation.
In robotics, planning refers to a group of algorithms that
take as input a goal state and produce a sequence of atomic
actions that will achieve it [8]. Task planning, a subset of
planning, enables robots to change the state of the objects
in the environment. A further subset of task planning is
proactive planning. This involves the active recognition of
context information and user intention when a goal is not
explicitly provided [9]. Most proactive planning approaches
are action oriented and infer goals primarily from user
behavior [10]. This view excludes scenarios where users
might not be able to act on the objects themselves, either
because of disabilities or because the items are in a remote
location. In addition, these applications rely on multiple
hypothesized goal states given a priori that typically cannot
be dynamically updated, which may impact performance
for a large goal space.
To overcome these restrictions, and inspired by the theory
of affordances [11], we developed (AP)2, an algorithm
that proactively identifies feasible sequences of atomic
actions by jointly considering objects and robot affordances.
Therefore, rather than inferring goals through recognizing
actions, (AP)2 considers the objects in the environment and
what robots afford over them to then infer feasible plans in
the environment.
To ensure that a large repertoire of plans is considered,
(AP)2 leverages large linguistic data sets that express plans
through human language, and it incorporates a search algorithm
to hierarchically organize these data sets based on
context features, e.g., where a plan can take place (locality)
and robot actions that are needed to complete it (capabilities).
A language-driven algorithm then retrieves the group
of plans from the data set that involves a context similar to
the one the user is in and ranks these schemes based on how
semantically similar they are when compared to the configuration
of the user's environment. Finally, (AP)2 applies a
filter to guarantee that only feasible plans are provided as
options, all of which are presented to the user using AR.
Our proposed algorithm leverages NLP methods to use
plans expressed through human language. NLP techniques
typically apply a mixture of symbolic (hand-designed features
and rules) and statistical (computed features and
learned rules) techniques to model language. Depending on
the techniques applied, these methods can be categorized as
1) classical NLP and 2) deep NLP, respectively [12]. Independent
of the techniques, both categories apply a standard pipeline
for cleaning and normalizing text. Text normalization
involves performing a series of tasks that include noise
removal (punctuation and special character deletion, numbers
and contraction replacement, and so on), lowercasing,
standardizing nearly identical words (coffee_machine: coffee
machine, coffee maker machine, coffee maker), out-ofvocabulary
word addressing, and stop-word elimination,
among others [12].
In comparison with the literature, we make the following
contributions:
●
●
We propose a method that enables an affordance-aware
AR HMD UI for robot control to combine atomic actions
and therefore provide higher-level options to the user.
We provide means for dynamically updating the goal states
that are considered and number of semantically relevant
plans that are analyzed. As a result, the time to provide
options to the user can be configured to a suitable value,
facilitating improved interactivity.
●
We validate the applicability of the proposed architecture
with an assistive mobile manipulator deployed in a bedroom
environment and demonstrate the comparable outputs
of (AP)2 against the proactive planning approach,
reporting the computational savings.
Related Work
Applications in the area of AR-based robot control can be
classified into two domains: remote interactions (i.e., teleoperation)
[13], where the user sends control commands
from outside the robot's location, and proximal interactions,
where the user and robot share the same space.
Applications in the latter domain include robot programming
[14], [15], trajectory planning [16], [17] and assistive
robotics [5], [6].
Most teleoperation applications overlay a video stream
from a remote location with virtual objects, such as a 3D
robot model and virtual handles that the user accesses
using either a computer display [18] or virtual reality
3
2
4
1
Figure 1. Given sensor information about objects in the
environment and the available robot capabilities, (AP)2
generates and ranks several plans to achieve potential tasks.
Current affordance-based AR HMD UIs for robot control require
at least four input commands to complete a task such as " pick
up the remote and put it on the bed " : (1) from the robot's
starting position, approach the TV stand, (2) pick up the remote,
(3) approach the bed, and (4) place the remote. Our platform
presents options by using human-like language via the AR UI to
complete the task through only a single command.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
23
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
