IEEE Robotics & Automation Magazine - March 2022 - 30

as input to the dense layer, and 3) using our word embedding
model as the embedding layer with the aggregated output of
all LSTM blocks. We chose to keep the latter, as it had the
smallest error during validation.
Finally, we implemented the feasibility filter by using an
off-the-shelf Fast-Forward v2.1 planner [40] and deployed
(AP)2 using Robot Operating System (ROS) Noetic on a
laptop computer with an Intel Core i7-8700K CPU and a
GeForce GTX 1070 mobile GPU. We included Microsoft
HoloLens 2 as the HMD and Unity 2019.4.10f1, Mixed
Reality Toolkit-Unity v2.5.3, and ROS Sharp to build the
AR HMD UI running on it. To achieve a stable display
framerate while dealing with long-running object detection
algorithms, we integrated our AR HMD UI with ROS via
ROS Sharp to run the algorithms on a separate computer.
For this, we format the photos taken with the HMD to make
them compatible with an appropriate ROS message and
publish them under a topic. Then, off-the-shelf object detection
algorithms, such as You Only Look Once ROS [41],
subscribe to this topic, process an image, and return the
resulting bounding boxes.
To demonstrate the applicability of the proposed architecture
in an AR HMD UI for robot control (Figure 1), we
set up a bedroom environment. The setup included receptacles,
e.g., a bed, chair, desk, and side table, and general
objects, such as a laptop, phone, TV remote, tennis racket,
and keychain. We used a Baxter robot plugged into a
Ridgeback omnidirectional base and equipped with two
Robotiq 2 F-85 grippers as assistive mobile manipulators.
We implemented navigation and pick-and-place capabilities
for Baxter using the affordance representation from (2).
The user context (8) that results from this setup is a =
" bedroom:navigation:pickplace. " In the " Results " section, we
report examples of options returned by (AP)2, the time from
an update of the state knowledge to the display of feasible
options in the HMD while changing the number of plans
considered after ranking, and the number of feasible plans
obtained. Further details of this demonstration are included
in the supplementary video available at https://doi.
org/10.1109/MRA.2021.3136789, provided by the authors.
Results
Figure 8 is a frequency distribution plot of the 30 most common
words in the data set to illustrate the effect of the NLP
cleaning and normalization. A total of 104,686 words were
included in the NIP
component when considering all E+
in
the data set's train split; 107 object classes were included in
the vocabulary, while the remaining were substituted by the
special word " PAD " because of their low frequency. Figure 4
includes the number of entries that resulted from the CD
added to the data set. Notice that, as all plans in ALFRED
require the robot to move between places for completion,
the algorithm does not add data set entries to the manipulation
nodes in the second hierarchy level of the tree. Therefore,
they are not shown in Figure 4. Only the navigation
node is further extended with all possible combinations of
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
Samples
Figure 8. The 30 most common words in the ALFRED data set's train split after cleaning and normalization.
30 * IEEE ROBOTICS & AUTOMATION MAGAZINE * MARCH 2022
Counts
Counter_Top
Sink
Microwave
Fridge
Table
Stove_Burner
Knife
Garbage_Can
Lamp
Cabinet
Shelf
Toilet
Sofa
Desk
Tomato
Chair
Bed
Potato
Apple
Coffee_Table
Plate
Lettuce
Dresser
Bowl
Bread
Coffee_Machine
Pan
Cup
Mug
Drawer

https://doi.org/10.1109/MRA.2021.3136789 https://doi.org/10.1109/MRA.2021.3136789

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022