IEEE Robotics & Automation Magazine - March 2022 - 28

OS
Sequence
Classification
Current Locality (LS)
+
{C0, ..., Cn} ⊆ Robot
Capability Labels (C)
User's Context
(α)
OS
Ranking
Algorithm
Semantically Ordered
Plans (ΨR) =
[(Pid1, DP1, GP1), ...,
(Pidm, DPm, GPm)]
Feasibility
Filter
kOS, ΩSl
Feasible Plans (P)
Figure 5. (AP)2 enables user-in-the-loop planning by identifying the most relevant and feasible robot affordances available to the user.
It integrates 1) a data set of plans expressed using human language and organized in a hierarchical context-dependent manner, 2) a
language-driven ranking algorithm to identify the most relevant plans given the user's context, and 3) a filter that determines whether
a plan is feasible.
where NIP
v
and O ,S
and OS
v
are the sentence embedding of NIP
respectively. We order the plan identifiers in W
WW3
R
from the highest to lowest similarity based on (9) and provide
as output a set of semantically ordered plans
.
Notice that often, RW will be a subset of W because, as we
show in our experiments, the time to provide options to
the user from an update of S increases proportionally with
the number of plans in
W Therefore, to keep this time
R .
reasonably short, the number of plans in RW needs to be
appropriately set.
S = kOS, ΩSl
ΨR =
[(Pid1, DP1, GP1), ...,
(Pidm, DPm, GPm)]
{ΛR/LS ∈ LR}
∀AR ∈ A
Feasibility Filter
At this stage, (AP)2 has a group of plans that potentially
involve objects and contexts that are semantically similar to
those from the user. However, we do not know if these plans
can be completed. (AP)2 uses the feasibility filter stage to
determine whether a plan is feasible. We built this stage
around a planning system. A planning system takes as input
a planning problem, which is a description of the initial
state, the actions available to change it, and the goal state [8],
and outputs a logic plan composed of robot actions that will
accomplish the objective when executed from the initial
state [37]. The input and output of the planning system are
expressed in an appropriate task planning syntax.
The various planning formalisms used in artificial intelliProblem
File
Instantiation
P =
[(Pid1, DP1, βP1), ...,
(Pidk, DPk, βPk)]
PDDL Planner
Figure 6. The feasibility filter. Our design leverages a PDDL
planning system, which expresses the planning problems using
the domain (actions) and problem (initial and goal states) files.
A planner takes these files as input and outputs a solution (if
any) in the form of an action sequence
b .
28 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
ΨR+
Domain File
Instantiation
gence have been systematized within a standard syntax called
the Planning Domain Definition Language (PDDL) [37]. The
PDDL divides the definition of a planning problem into two
files: the domain file, which defines actions, and the problem
file, which is a specific planning problem instance that
defines the initial state and a goal state. A planner takes these
files as input and outputs a solution in the form of an action
sequence. Even though many other task planning languages
exist [37], we chose the PDDL to implement (AP)2, as we
found it suitable to answer the central question we are interested
in: whether a sequence composed of available low-level
robot actions succeeds in reaching a goal from the current
state or not [7]. Our design is detailed in Figure 6.
()
AP 2
dynamically instantiates a domain file by incorpoK
;! 6 ! Then, the algorithm
RS RR
LA
(3)
rating the set of robot actions that can be applied in the current
locality; i.e., LA{ }.
takes each entry in RW and complements its associated GP
ΛR Could be Useful (LR)} ∀ Robot Affordance (AR) ∈ A
{Set of Robot Action (ΛR)/LS ∈ Localities Under Which
{Set of Robot Action (
ΛR Could be Useful (LR)} ∀ Robot Affordance (AR) ∈ A
Relevant Plans (Ψ) =
[(Pid1, DP1, NIPn
(Pidn, DPn, NIPn
Search (Γ)
, GP1), ...,
,GPn)]
E+ = kPlan id (Pid),
Plan Description (DP),
Plan Instructions (IP),
Goal States (GP),
)l
Data Set
of Human-Like
Plans (Φ)
Robot Affordance
Set (A)
Context Declarations (CD),
Normalized IP (NIP
S = kObject Classes (OS),
Spatial Coordinates (ΩS)l
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
