IEEE Robotics & Automation Magazine - June 2019 - 51

psychotherapists, engineers, and ethicists) have been involved
throughout the system development process in a concurrent
manner. A robotic system used in RET should meet the
requirements from both therapeutic and robotic perspectives.
Key elements of the requirements are illustrated in Figure 2
and summarized in this section as follows [11].
First, the system should enable the robot to generate taskbased social behaviors to achieve therapeutic goals, which is
the ultimate goal of using robots in therapeutic contexts. Second, the robot control should be shared with human therapists
to ensure safe and ethical behaviors. Third, the system should
be applicable to various therapeutic scenarios and robot platforms that reduce engineering workload, e.g., reprogramming
a robot's actions. Lastly, the system should analyze data (e.g.,
the user's performance history and robot operation) recorded
in structured forms and provide it to different parties.
These requirements serve as guidelines and evaluation criteria for RET systems. Some of the system design principles
used to obtain these requirements are 1) multilayered behavioral organization (for generating task-based and social
behaviors), 2) personalization (for providing personalized
interaction), and modularity (to help with applying the system to different scenarios and robot platforms [12]). During
development of the DREAM project, we adopted some of
these design principles to establish a supervised autonomous
system for different tasks in autism therapy (see the "Supervised Autonomous System" section).
Clinical Framework
To assess socially assistive robots' effectiveness in enhancing
social skills in children with ASDs, certain behaviors have
been frequently targeted by therapeutic interventions. Among
them (and for the specific goal of the DREAM project), we
have targeted the following behaviors: imitation, turn taking,
and joint attention. These behaviors, including communication and social interaction deficits, could be considered as
possible mechanisms that underlie the general clinical picture
and will be taught by a social robot during repeated therapy
sessions of interactive games.
Supervised Autonomous System
Maneuvering a robot to deliver a therapy is a complex task,
and in the case of supervised autonomous RET, it requires
procedures that 1) sense the state and performance of the
child and 2) select and execute an action for the robot
(according to a therapeutic plan) while providing oversight
of the robot's behavior to the therapist. This process is
engineered by an interconnected network of components,
as shown in Figure 3. These components are responsible
for sensing and interpreting the surrounding environment,
classifying a child's behavior, and controlling robot behavior. The system also provides an intuitive graphical user
interface (GUI), which allows the therapist to supervise
the system operation and ensures efficient robot behavior.
All system of the components were released [28] under the
GNU General Public License v3 and documented, which

allows researchers to replicate, modify, or expand the
DREAM system for different target applications.
Sensory System
An advanced sensory system translates multisensory data into
meaningful information about the child-robot interaction, e.g.,
a child's movement, gaze, vocal prosody, emotion expression,
and typical ASD behaviors. Different techniques have been
applied to raw images captured by red, green, blue (RGB) cameras and Microsoft's Kinect sensors for gaze estimation, skeleton joint-based action recognition, face and facial expression
recognition, object tracking, and audio data processing.
Gaze estimation is vitally important for identifying shared
attention in child-robot interactions during joint-attention
tasks. The challenges to gaze estimation that emerge during
therapeutic sessions are related to head movement, illumination variation, and eyelid occlusion. Feature points on the face
are located by a supervised descent method, which is based on
the best view of the child's face. The head pose is calculated by
an object pose-estimation method. Iris centers are localized by
a hierarchical adaptive-convolution method [see the red dots
in Figure 4(a)]. The final gaze point is calculated based on the
obtained head pose and iris centers by a two-eye modelbased method [see the white line in Figure 4(a)] [13].
Human action recognition, i.e., a child's actions during
interaction, plays a key role in evaluating imitation tasks performed by the child. A novel skeleton joint descriptor that uses
a 3D moving trend and geometry property is applied on skeleton data extracted from Kinect's depth sensors [Figure 4(b)]
[14]. The descriptor is then used to recognize actions (e.g., waving, touching the head with two hands, moving the arms to
imitate an airplane, or covering the eyes) by a linear support
vector machine (SVM) classification algorithm.
Facial expression recognition provides an understanding of
the child's emotions, e.g., anger, disgust, fear, happiness, sadness, and surprise. This is achieved by using a frontalization
method to recover frontal facial appearances from unconstrained nonfrontal facial images, followed by a local binary
patterns feature-extraction method applied to three orthogonal planes to represent facial appearance cues. Finally,
we applied an SVM to identify and classify those facial
Goals

Therapist

Scenarios

Sensors

Data

Robot Platforms
Figure 2. The elements a RET robotic system should consider for
generating robot behaviors [11].

JUNE 2019

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2019