IEEE Robotics & Automation Magazine - March 2022 - 25

placed, such as beds, desks, and night tables, among others,
and that generally have a fixed position in the environment,
and 2) general signifiers, which represent objects for which
we cannot make assumptions about positions ahead of time.
Receptacles have predefined properties: their 1) object
class label and 2) position in the environment. The latter is
affixed to the real environment via spatial anchors, i.e., a spatially
and temporally stable, six-degree-of-freedom pose relative
to visual features provided by simultaneous
localization and mapping system of the HMD [30]. This
makes receptacles stay precisely in place between runs of
the application. Predefined properties are defined and
stored in the HMD. General signifiers, instead, have no
predefined properties. Therefore, their object class and
position in the environment are obtained at runtime, using
the object detection pipeline illustrated in Figure 3 [5].
In addition to the class and position of each object, general
signifiers provide information about the receptacle on
which the object is placed. For this, we apply ray casting, a
process that projects a ray into the environment, returning a
Boolean value if a target is hit. When this happens, information
about the hit, such as the distance and position as well
as a reference to the object's transform, can be stored in a ray
cast hit variable for further use. General signifiers ray cast a
ray from their position in the direction of gravity and obtain
information about their corresponding receptacle from the
hit. This information is useful when robots need to manipulate
an object. Rather than navigating directly to an object's
position, robots navigate to the associated receptacle and use
the object's position only for placing and grasping.
Colocalization
Colocalization is the process that enables localizing the
robots and a HMD in the same global coordinate system.
The poses included in S are relative to the HMD's frame of
reference and cannot be directly used by the robots. To find
the transformation between their frames, we use a common
map approach that applies singular value decomposition to
the poses of virtual markers equipped with spatial anchors
and their equivalent coordinates on a 2D occupancy grid
map for robot localization [31].
Data Set
To leverage knowledge about how humans combine
objects to accomplish their goals, rather than creating a
library of predefined plans with logical descriptions of the
actions robots need to perform, we considered publicly
available data sets of plans given using human language as
the data set of plans ()U for (AP)2. We express the minimum
information that each of the data set entries (E)
needs to provide as
EPid ,, ,,DI GPP P
where Pid
description, IP
is a unique identifier for the plan, DP
=GH (3)
is the plan
represents the plan instructions, and GP
is
the goal state that expresses the plan description in an
appropriate task planning language. Data sets that incorporate
this kind of information include Recipe1M+ [32],
which provides more than 1 million recipes ()IP
into 1,000-plus categories (),DP
divided
and Action Learning From
Realistic Environments and Directives (ALFRED) [33],
which consists of 25,743 human language directives that
include a DP
and a set of IP
per entry.
We extended the information provided by U by adding
I ).P
two new fields to each of the variable's entries: context declarations
(CD) and normalized plan instructions N(
Here, CD are text strings that summarize environmental
features, such as locality and robot capabilities:
CD ::: :,
= LC CCPP PPn01
g
where L LP
(4)
! is the locality label that represents the location
for which the associated plan is relevant and
Camera View
Space
Take
Picture
2D Position
App
Approach
Object
Picture
(a)
Detection
Coordinate
System
Camera
Coordinate
System
(b)
(c)
Figure 3. The object detection pipeline. (a) The user takes pictures via natural ways of interaction, such as hand gestures and voice
commands. These pictures are processed through an object detection model running on a separate computer to obtain the 2D
position and class of identified objects. (b) With this information, the HMD performs a camera-to-application-coordinate-system
transformation to (c) place the signifiers associated with the identified objects in the 3D environment.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
25
Bottle
Cup
Camera
Projection
Space
Screen
Keyboard

IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022