IEEE Robotics & Automation Magazine - June 2019 - 87

experiment, the prompt modalities are
Reasoning and Decision Making
an augmented version of the uploaded
* Knowledge Model
Perception
image as well as the audio. Live video
* Machine Learning
* Interpret the Context
augmentation is outside the performance abilities of our hardware implementation but could be included in
Online
Sensing
future work.
Action
Process
* Voice
To provide this instruction, the sysSelection
* Photo
tem parses the image for relevant
information before applying supervised learning to solve the problem of
identifying the correct context of the
Interaction
image. Classifier output combined
Reality Augmentation
* "What's Next?"
with the task's knowledge model allows
* Voice
* Images
the selection of the correct prompt for
+
* Video
the next step in the task, which is delivered seamlessly through the AR interface to the student.
Wearable
Correct
Device
Our AR system implementation
Behavior
consisted of a Google Glass wearable
AR device running a simple Android
"What do I
application with a streamlined interdo next?"
face. The simple audio command
"Okay Glass, what's next?" triggers the
app. The user clicks to take a picture, Figure 10. The AR system overview. The student is instructed to face the device and ask
which is uploaded to the cloud server, for assistance. A picture taken from the wearable camera is classified, and an augmented
and an image and audio instructional image and audio containing the proper instructional prompt for the next step are delivered.
prompt are provided within 5-10 s via
the Glass display and built-in speaker. On the cloud side, account statement skill (an employment and independent livintelligent instruction is made possible using OpenCV for ing skill), the student was asked to log in and retrieve a copy of
image processing and support vector machines trained on his or her prepaid student account statement. The third experextracted visual features for image classification. In the event iment, geometric reasoning, taught object manipulation,
of a failed or low-probability classification, the user is present- placement, orientation, and assembly subskills that are highly
ed with a prompt to try again.
vocationally relevant.
Three skills were taught using the AR device: 1) using a
For these experiments, instructional strategies were carecopy machine, 2) accessing one's student account statement fully designed using the appropriate combination of prompts,
online, and 3) performing geometric reasoning tasks with prompting strategies, and prompt chaining for each skill
puzzle blocks to acquire the prerequisite ablity for the taught and the AR technology method (Table 2). These comadvanced skills taught by the IRI system. Figure 11 shows binations of methods were selected with expert consultation
example annotated image prompts from these experiments. A to match the appropriate prompt strategy and chaining direcsummary of these experiments appeared in [16].
tion to the structure of the skill.
For the copy machine skill (an office vocational work skill),
All three AR experiments used a multiple-baseline-acrossstudents were asked to make a specific number of double-sid- skills design, in which each subskill or step was mastered
ed copies of a document on a commercial copy machine with before the successive one was introduced. Also, in all three,
a complex, less-than-intuitive user interface. For the student the skills were taught in forward-chained order because the

Table 2. The experiment overview.
Platform

Experiment

Prompt

Design

Relationship

ARI

Copy machine

Self-directed

Multiple baseline across skills

Forward chaining

Student account

Self-directed

Multiple baseline across skills

Forward chaining

Geometric reasoning

CTD

Multiple baseline across skills

Discrete and forward chaining

Making change

SLP

Multiple baseline across participants

Discrete and forward chaining

Geometric assembly

SMP

Multiple baseline across skills

Backward chaining

IRI

JUNE 2019

IEEE ROBOTICS & AUTOMATION MAGAZINE

IEEE Robotics & Automation Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - June 2019