IEEE Robotics & Automation Magazine - March 2022 - 33

fewer or no options were provided for
these objects.
Bed
Conclusions
In this article, we presented (AP)2, an
algorithm that proactively identifies
feasible sequences of atomic actions by
leveraging large data sets of plans
defined using human language. (AP)2
incorporates language-driven and
planning algorithms to provide the
most relevant and feasible schemes
given the user's context and provides
means to reduce the time required to
generate and present them as options
to the user. We introduced a method
that enables an affordance-aware AR
UI for robot control to combine atomic
actions and generate higher-level
options to the user. Furthermore, we
provided means for dynamically updating goal states and the
number of semantically relevant plans that are analyzed to
facilitate ways to improve interactivity for the user. We also
validated the applicability of the proposed architecture with
an assistive mobile manipulator deployed in a bedroom environment
and controlled using an AR UI.
However, our implementation relies on written English
Book (618)
Tennis_Racket (118)
Remote_Control (665)
Laptop (598)
Key_Chain (835)
Cell_Phone (721)
100
(1,690)
12
6
Arm_Chair
(174)
5
5
3
9
2
200
400
Desk
(2,029)
10
13
6
Dresser
(1,118)
3
5
2
Side_Table
(802)
9
6
2
2
500
3
631
Figure 11. The combination matrix for receptacles and general objects. Colors represent
the number of plans the feasibility filter analyzed before outputting an option describing
the combination of low-level actions, general objects, and receptacles. The frequency of
appearance in the data set for each associated word and the number of options resulting
in the same combination are also shown.
text to present robot affordances to the user. Text-based menu
options are limited in effectiveness to those who can read and
are effective only for those who understand the language
being used. This limits our platform from reaching a more
diverse user base. Furthermore, our implementation presents
the set of viable plans in the same order for every scheme output
by the feasibility filter. Such a set could be big, and finding
the option the user is actually interested in might take time,
potentially limiting how useful our platform will be perceived
as being. In future work, we will design and evaluate a series
of graphic devices to communicate information without
words. In addition, we will incorporate the user's preferences
and habits as part of the context information employed during
ranking to make its order more relevant.
Acknowledgments
Rodrigo Chacón Quesada is supported by a scholarship from
the University of Costa Rica and the Ministry of Science
Technology and Telecommunications, Costa Rica. Yiannis
Demiris is supported by a Royal Academy of Engineering
chair in emerging technologies. This research was supported,
in part, by UK Research and Innovation grant EP/V026682/1.
This article has supplementary material available at https://
doi.org/10.1109/MRA.2021.3136789, provided by the authors.
References
[1] Y. Demiris, " Knowing when to assist: Developmental issues in lifelong
assistive robotics, " in Proc. 2009 IEEE Annu. Int. Conf. Eng. Med.
Biol. Soc., pp. 3357-3360, doi: 10.1109/IEMBS.2009.5333182.
[2] D. Feil-Seifer and M. J. Mataric, " Defining socially assistive robotics, "
in Proc. 9th 2005 IEEE Int. Conf. Rehabil. Robot., ICORR, pp. 465-
468, doi: 10.1109/ICORR.2005.1501143.
[3] L. Jamone et al., " Affordances in psychology, neuroscience, and
robotics: A survey, " IEEE Trans. Cogn. Devel. Syst., vol. 10, no. 1, pp. 4-25,
Mar. 2018, doi: 10.1109/TCDS.2016.2594134.
[4] F.-J. Chu, R. Xu, Z. Zhang, P. A. Vela, and M. Ghovanloo, " Handsfree
assistive manipulator using augmented reality and tongue drive
system, " in Proc. 2018 IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS),
pp. 5463-5468, doi: 10.1109/IROS.2018.8594508.
[5] R. Chacon and Y. Demiris, " Augmented reality controlled smart
wheelchair using dynamic signifiers for affordance representation, " in
Proc. 2019 IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), pp. 4812-
4818, doi: 10.1109/IROS40897.2019.8968290.
[6] R. Chacon and Y. Demiris, " Augmented reality user interfaces for
heterogeneous multirobot control, " in Proc. 2020 IEEE/RSJ Int. Conf.
Intell. Robots Syst. (IROS), pp. 11,439-11,444, doi: 10.1109/IROS45743.
2020.9341422.
[7] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez, " From skills to
symbols: Learning symbolic representations for abstract high-level
planning, " J. Artif. Intell. Res., vol. 61, no. 1, pp. 215-289, 2018, doi:
10.1613/jair.5575.
[8] S. Russel and P. Norvig, Artificial Intelligence: A Modern Approach.
London, U.K.: Pearson Education, 2013.
[9] A. J. Schmid, O. Weede, and H. Worn, " Proactive robot task selection
given a human intention estimate, " in Proc. 2007 RO-MAN 16th IEEE
Int. Symp. Robot Human Interact. Commun., pp. 726-731, doi: 10.1109/
ROMAN.2007.4415181.
[10] H. Harman and P. Simoens, " Action graphs for proactive robot
assistance in smart environments, " J. Ambient Intell. Smart Environments,
vol. 12, no. 2, pp. 1-21, 2020, doi: 10.3233/AIS-200556.
[11] J. Gibson, The Ecological Approach to Visual Perception. Boston,
MA, USA: Houghton Mifflin, 1979.
[12] D. Jurafsky and J. H. Martin, Speech and Language Processing: An
Introduction to Speech Recognition, Computational Linguistics and Natural
Language Processing. Upper Saddle River, NJ, USA: Prentice Hall, 2008.
MARCH 2022 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
33
1
3
https://doi.org/10.1109/MRA.2021.3136789 https://doi.org/10.1109/MRA.2021.3136789
IEEE Robotics & Automation Magazine - March 2022

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2022
