IEEE Robotics & Automation Magazine - March 2022 - 29

with the object classes and positions provided by S to build a
problem file for it. We denote this new set of entries augmented
with problem files as
PDDL-capable planner to determine which of the plans in
R
W +
P R
together with the logic sequence of robot actions
are feasible. If the planner finds a solution for a particuAP
2
lar plan, () includes the scheme in a set of feasible plans
3 W +
required for it ().b Here, P is the output of (AP)2; we use it to
provide afforded options to the user through our AR HMD
UI by presenting the corresponding plan descriptions [DP
in
(3)] for each plan in P. When the user selects an option, the
associated b is moved to the execution phase, where it is dispatched.
In Figure 7, we demonstrate how the user can access,
inspect, and select the feasible plans included in P.
Experiments
Unlike current proactive planning methods, which require
processing an entire data set during a search, (AP)2 dichotomizes
the search space and dynamically updates potential
goal states. The general aim of this experiment is to demonstrate
the comparable outputs between (AP)2 and current
proactive planning methods and report the associated computational
saving. In addition, we show the applicability of
the proposed algorithm in an affordance-based AR HMD UI
for an assistive robot control scenario.
We used the ALFRED data set as our library of plans for
this experiment [33]. ALFRED combines four possible locations:
kitchen, living room, bathroom, and bedroom, together
with navigation actions to move the agent and change its
camera orientation as well as manipulations for picking and
placing objects, opening and closing cabinets and drawers,
turning appliances on and off and slicing objects. The 8,051
data set entries in ALFRED are divided into training, validation,
and test splits.
To begin, we defined L {,
= kitchen livingroom, bedroom,
bathroom}) and C {,= navigation pick_place, open_
close, turn_on_turn_off, slice} in accordance to ALFRED
W +R . We then use an off-the-shelf
features. This results in the search tree structure in Figure 4.
Next, we added CD (4) to each E via a script using metadata
included in ALFRED. The script obtains LP
by checking the
scene number, which determines the location for the plan.
For example, scene numbers 1 to 30 correspond to kitchen
environments. Then, the script obtains the robot capability
labels by checking the plan metadata from which the navigation
and manipulation actions required to complete the plan
can be obtained.
Then, we continued by adding NIP
to each E by using
I .P There are 125
standard NLP cleaning and normalization methods to
extract the object classes from their
object classes in ALFRED, and we used them as our starting
point to define our set of valid words, i.e., vocabulary.
To facilitate learning an appropriate embedding, and following
the common practice for relatively small vocabularies
such as ours when dealing with out-of-vocabulary
words, we used a special word ( " PAD " ) to substitute object
classes with fewer than five occurrences in the data set [38].
Nevertheless, the previously described method is known to
be applicable and generalize to larger vocabulary sizes [39].
The script could include words in NIP
only if they were
part of our vocabulary.
After augmenting each data set entry in ALFRED with CD
we used the latter to train our word embedding
model. For this, we employed the Word2Vec module from
Gensim library 3.8.3., with 25 as the embedding dimension, a
window size of three, and skip-gram representation. Next, we
implemented the sequence classification model using Microsoft
Cognitive Toolkit 2.7 and trained it with the Adam optimizer,
using a batch size of 64 for five epochs. We used 25 as
the embedding dimension, 50 as the dimension for the LSTM
cell, and four as the number of output classes for the dense
layer, one per locality label in L. We compared three model
configurations: 1) linear embedding and the final LSTM
block for classification of the entire sequence, 2) linear
embedding and the aggregated output from all LSTM blocks
and N ,IP
(a)
(b)
(c)
(d)
Figure 7. The feasible plans provided as output by (AP)2 are presented to the user by using AR. The background images show the
bedroom setup we used during the experiment.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
29

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022