IEEE Robotics & Automation Magazine - March 2022 - 24

HMD [19]. Proximal interactions typically employ a computer
screen or an AR HMD to overlay the environment
with virtual elements, such as the planned motion of a
robot [17], the torques experienced by the joints of a
robotic arm [20], and virtual buttons [15]. Input modalities
for remote and proximal interactions include touchscreens
[21], hand gestures [15], [17], [22], fiducial
markers [23], haptics [24], hand-held devices [14], [25],
electroencephalography signals [26], eye gazes [27], and a
combination of these [5], [6], [28].
However, most of the previous work associates AR-based
robot control with low-level instructions, limiting interaction
to what more traditional control methods, such as keyboards
and joysticks, provide. When multiple instructions can be
provided at once, they are limited to the combination of the
ones that can be simultaneously applied [6], resulting in long
sequences of commands, such as in Figure 1. To overcome
these limitations, we propose an architecture that integrates
an AR HMD UI with a language-driven algorithm to suggest
and execute tasks, such as " pick up the remote, and put it on
the bed, " that are more complex in terms of their sequence
length and action space.
(AP)2
In this section, we introduce our algorithm to infer the feasible
plans available to the user, given the observed objects and
robot affordances in the environment. An overview of the
method is presented in Figure 2.
Environment Representation
We provide information about the environment to (AP)2
by using an AR affordance-based UI [5], [6]. This UI
dynamically updates the state knowledge by providing
the spatial coordinates and object classes of items identified
in the environment, as illustrated in Figure 3. We use
a data structure called the current state (S) to represent
this knowledge:
S O ,,= X H
G SS
where OS
(1)
is the set of object class labels and SX denotes
their corresponding 3D poses. We represent the ways in
which a robot can interact with objects in the environment
by using robot affordances ().AR
This resembles the formalization
of affordances from an observer's perspective,
used when the interaction between an agent and the environment
is observed by a third party [29], e.g., a user wearing
an HMD [6].
Given a predefined set of locality labels L( ), e.g.,
L {,= bedroom office, backyard,...} and a predefined set of
robot capability labels C( ), e.g., C {,= navigation pick,
place,...}, robot affordances can be expressed as
AR ,, OL C,, ,RR RR R
= K
G
where AR
R
id
H
(2)
represents the robot affordances of agent R ;id
K is the set of actions the robot can perform (along with
preconditions and effects), expressed
in an appropriate task planning language;
and OR
3 and CCR
is the set of objects
3 are used
Objects
Sensory
Input
(Pictures,
Spatial Anchors)
State
Knowledge
(Spatial
Coordinates,
Object Classes)
Options
Feasibility
Filter
(b)
(c)
Figure 2. (AP)2 identifies the most relevant and reasonable plans available to the user
by jointly considering objects and robot affordances. It integrates 1) a data set of plans
expressed using human language, 2) a search algorithm that organizes the data set
in a hierarchical context-dependent manner, 3) a language-driven ranking algorithm
to identify the most relevant plans given the user's context, and 4) a feasibility filter
to determine whether or not a plan is reasonable. (a) The environment. (b) The AR
affordance-based UI. (c) (AP)2.
24 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
Ranking
Search
(a)
Robot Affordances
Robots
K could be
= f
RRn () uses
AP 2
across which RK can be applied.
Here, LLR
to summarize the context of the
environment in which R
useful. In multirobot settings, we
represent a set of robot affordances
as A {, ,}.AA
robot affordances as we describe in
the " Search " section.
Data Set of
Human-Like
Plans
Signifiers
The information included in S is
obtained by the UI from the signifiers
placed in the environment [Figure
3(c)]. Signifiers are defined as
signs, symbols, marks, sounds, and
any other perceivable indicator that
conveys meaningful information to
explain what actions are possible, how
they should be executed, and where an
interaction should take place [5]. We
categorized signifiers into two general
classes depending on their functionality:
1) receptacles, which signify
objects on which other items can be
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
