IEEE Robotics & Automation Magazine - March 2013 - 37

inspired us to develop software capabilities and user interfaces that support social self-expression by people with
both motor and speech difficulties (as can also occur for
people with ALS [1]). Our goal is to allow a person to communicate and socially interact in a more satisfying way.
In this section, we discuss two interfaces that we have
developed to facilitate social communication.
Audio Support
The first interface is called SpeakEasy and is shown in
Figure 12. SpeakEasy allows a user to effectively control multiple speech engines simultaneously, for example, controlling
one engine onboard the robot and another on the user's laptop.
SpeakEasy is agnostic about the brands of the TTS engines.
The graphical interface is divided into five sections. Basic
functionality is in the text entry section, which receives text
entered via an onscreen keyboard, and the tape recorder buttons, which transmit the text.
To make the interface more efficient, the utterance program bank allows user-labeled buttons to be programmed
with text of any length. The bottom row of the interface (see
Figure 12) provides a method for storing and easily retrieving
sets of buttons of preprogrammed utterances.
To make the interface more effective and interesting, the
sound bank section stores sound effects that can be broadcast
to standard audio players in multiple locations at once.
A practical use of sound effects is using a warning sound
when the robot drives around blind corners. Henry requested
a more amusing sound effect; he wanted a powerful glasspack
muffler sound for his electric wheelchair. This shows once
again that utility is best measured by the user.
Creating Gestures with RCommander
Early on in the project, Henry explicitly expressed interest in
performing a standup comedy routine using the PR2 as a
proxy for his body. In spite of years of research with AMMs,
this was not a task that we had previously considered.
Gestural expressions can play an important role in standup
comedy and in social interactions in general. Gestures can
also be diverse, distinctive, and personal. As such, we wanted
to empower people to create their own gestures, reuse those
gestures, and share and adapt gestures created by other users
in their communities.
With these goals in mind, we modified the RCommander
robot behavior editing tool seen in Figure 13 to serve as a gesture-authoring tool. RCommander is a general tool that we are
developing to enable nonroboticists like Henry to create their
own behaviors for robots. As such, creating custom gestures
has served as a first test case for enabling a person with severe
motor impairments to create robot behaviors for himself.
Each set of robot actions that accomplishes a task is represented as a state machine with each state drawn as a circle
representing one of a robot action. RCommander enables
users to create new gestures through an iterative modifyand-test process, save gestures in a library, and create
sequences of actions. The two classes of actions that Henry

Figure 11. Henry and Jane use a communication board.

Figure 12. The SpeakEasy interface allows the user to
control speech and sound onboard the robot and elsewhere.
(Retouched for reproduction.)

tested were splined trajectories for designing robot gestures
and TTS generation.
The gesture GUI is structured as a keyframe animation
tool in which interactive markers are used to create the keyframe poses. After a keyframe is posed, RCommander assists
the user in debugging by highlighting the joint angles that are
too close to the joint limits or that violate velocity constraints.
Using interactive markers instead of physically posing the
robot (as is typically done in learning from demonstration
research), makes our interface accessible to Henry.
Henry has tested our interface multiple times. Each test
begins with a tutorial, followed by an exploration period in
which Henry has constructed gestures using a PR2's arms.
After each session, we applied the lessons learned to iterate
on the interface. For example, our sessions have highlighted
the importance of quick access to behaviors during use,
March 2013

IEEE ROBOTICS & AUTOMATION MAGAZINE

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2013