IEEE Robotics & Automation Magazine - June 2019 - 41

defiant, in most cases they are likely a result of inadequate
management of emotions in response to stress or overstimulation [6].
Traditionally, therapies targeting ER in ASD have revolved
mainly around cognitive behavioral therapy, which specifically targets co-occurring anxiety in children with ASD [7].
However, more recent research efforts [8] have focused on
therapies that use techniques of mindfulness and meditation
to cultivate an increased awareness of one's internal reactions
and emotions to utilize during times of distress. While a
plethora of previous studies with SARs have developed and
evaluated solutions to alleviate the social [9], language [10],
and learning [11] impairments associated with ASD, not as
much attention has been focused on empathy or ER. This is
noteworthy given that impairment in these two core areas
may be the underlying cause of many of the behavioral disturbances commonly reported in the disorder. To the best of our
knowledge, there has been only one study involving SARs
that directly targets ER in ASD, but it is currently in its preliminary stages [12].
In our work, we developed a framework that implements
empathy-based ER while engaging the user in turn-based
emotional interactions. To this end, we chose a penguin character to design a friendly, emotionally expressive agent that
communicates emotions. The animated character was also
deployed to a mobile iPod-based robot to enhance its emotional expression capabilities.
In addition, we developed a framework for children
with ASD to engage in sensory interactions. A sensory
station activity was designed for this purpose [13]. It utilized both Romo and a small humanoid robot, which are
presented with various forms of sensory stimulation as
they walk through a sensory station setup. The robots
modeled socially appropriate responses to sensory stimuli
and attempted to teach the children to communicate their
feelings effectively so that a meltdown or tantrum resulting from overstimulation could be avoided. Although this
framework is not the focus of the current work, the relevant details of this study are discussed in "User Study"
and "Experimental Results."
Designing an Emotionally Expressive Agent
The original Romo, developed by Romotive and shown in
Figure 1, came with a blue monster-like character capable of
articulating emotion through the display of a variety of facial
expressions. The iPod was mounted on treads that allowed it
to accompany the facial expressions and sounds with physical
movements. However, to gain full control of the system, we
decided to design our own character and software framework. We replaced the monster with a penguin character, and
several factors were taken into account to ensure an ASDfriendly design, as described in [14]. The design principles
consisted of the following.
● Any verbal expressions must be kept simple and short.
● Avoid using complex patterned background images.
● Avoid horizontal scroll bars.

Background music must be avoided whenever possible to
minimize distractions.
● Minimize the use of flashing or moving content.
● Use a simple spatial layout, with minimal information display on the screen.
The color selections for the design of the character were
also carefully considered according to the principles presented in [15]. The color palette for our character was based
on the principles detailed in [14] and [15], with a focus on
muted colors to minimize any possible discomfort caused to
a child with ASD.
The next step was to
determine the appropriIn our work, we developed
ate color for each emotion. A set of 14 emotions
a framework that
was selected; each represented a variation in intenimplements empathy-based
sity level for the six basic
emotions (anger, disgust,
ER while engaging the user
fear, happiness, sadness,
and surprise). For examin turn-based emotional
ple, "content," "happy,"
and "excited" representinteractions.
ed progressively increasing intensities of the
happiness emotion. A suitable emotion color mapping was
determined using the emotions color wheel [16].
To design the facial expression of the penguin character,
we considered several samples from Greive [17]. Based on
these sample emotion cards and inputs from several parents of children with ASD, the expression design of the
penguin character consists of four distinct layers: text
●

(a)

(b)

Figure 1. (a) The original Romo and (b) our custom-designed
Romo.

JUNE 2019

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2019