IEEE Robotics & Automation Magazine - March 2013 - 58

sensorimotor data in an online, interactive setting, these participants worked with the robotic platform to complete a
series of randomized actuation tasks. Participants actuated
one of the robot's arms using a conventional myoelectric
controller with linear proportional mapping. The EMG signals were sampled and processed according to standard procedures from four input electrodes affixed to the biceps,
deltoid, wrist flexors, and wrist extensors of a participant's
dominant arm. Pairs of processed signals were then mapped
into velocity control commands for the robot's elbow roll
actuator and hand open/close actuator. In each task, one arm
of the Nao robot was moved to display a new gesture consisting of a static angular position on both hand and elbow actuators. Subjects were asked to make a corresponding gesture
with the robot arm under their control. Once a subject maintained the correct position for a period of more than 2 s, a
new (random) target configuration was displayed. Visual
feedback to participants consisted of a front-on view of the
robot system. Subjects performed multiple sessions of the
randomized actuation task, with each session lasting between
5 and 10 min. No subject-specific changes to the learning
system were made; all subjects used exactly the same learning system with the same learning parameters, set in advance
of the trials. To assess the real-time adaptation of learned
predictions, additional testing was done via longer unstructured interaction sessions, some of which lasted over 1 h and
included tasks that produced moderate muscle fatigue.

As depicted in Figure 2, the learning system observed the
stream of data passing between the human, the myoelectric
controller, and the robot arm. We created two GVFs for each
of the different signals of interest rq in the robotic
system-one to predict temporally extended signal outcomes
at a short time scale (+0.8 s), and one to predict outcomes at a
slightly longer time scale (+2.5 s). As done in previous work
[7], the learning system was presented with a signal space consisting of robot joint angles and processed EMG signals; at
every time step, the function approximation routine mapped
these signals into the binary feature vector x l used by the
learning system. All signals were normalized between 0.0
and 1.0 according to their maximum possible ranges.
Parameters used in the TD learning process were
m = 0.999, c = {0.97, 0.99}, and a = 0.033. Weight vectors
e and w for each GVF were initialized to 0. Learning updates
occurred at 40 Hz, and all processing related to learning, EMG
acquisition, and robot control was done on a single MacBook
Pro 2.53 GHz Intel Core 2 Duo laptop.

58

*

IEEE ROBOTICS & AUTOMATION MAGAZINE

Return

EMG Signal

Return

Angle (Radians)

Results
We found that predictions learned using our GVF approach
successfully anticipated measured signals after only short periods of online learning. Figure 4(a) shows an example of joint
angle prediction for the 0.8-s time scale with one subject after
+10 min of learning. Here, changes to the normalized return
prediction signals Pr q for both the hand (solid red trace) and
elbow (dotted red trace) joints can be
seen to occur in advance of changes
to the measured actuator signals (grey
lines). Predictions for both joints can
40
Predictions
1.0
Hand
be seen to precede actual joint activity
20
0.5
by 0.5-2.0 s. The system was also able
0
0.0
Predicted
to accurately predict myoelectric sigReturns
-0.5
nals [Figure 4(c)]. Normalized EMG
-20
predictions (red line) rise visibly in
-1.0
True
-40
advance of the actual myoelectric
-1.5
Returns
Elbow
-60
events (grey line), and changes to the
513 516 519 522 525 528 531
516 519 522 525 528 531 534
processed myoelectric signal were
Time (s)
Time (s)
anticipated up to 1,500 ms before
(a)
(b)
change actually occurred. The accu3.0
60
True
racy of predictions for both actuator
Anticipation
Predicted
Returns
Predictions
2.5
50
and myoelectric signals can be seen in
Returns
EMG
2.0
Figure 4(b), and (d). For both slow
40
1.5
and fast changes in the signal of inter30
1.0
est, the return prediction ( Pq , blue
20
0.5
line) largely matched the true return
10
0.0
as computed post hoc (R q , grey line),
0
indicating similarity between learned
535 540 545 550 555 560 565
540 545 550 555 560 565
predictions and computed returns.
Time (s)
Time (s)
As shown in Figure 5, accurate
(d)
(c)
predictions could be formed in 5 min
or less of real-time learning. These
Figure 4. Examples of (a) and (b) actuator and (c) and (d) myoelectric signal prediction during
able-bodied subject trials after +10 min of online learning. (a) and (c) Normalized return
learning curves show the relationship
predictions (red traces) precede the observed signal activity (grey lines) by 0.5-2.0 s. (b) and
between prediction error and train(d) Return predictions (blue traces) are consistent with the true return as computed post hoc
ing time, as averaged across multiple
(grey lines).
*

March 2013



Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2013

IEEE Robotics & Automation Magazine - March 2013 - Cover1
IEEE Robotics & Automation Magazine - March 2013 - Cover2
IEEE Robotics & Automation Magazine - March 2013 - 1
IEEE Robotics & Automation Magazine - March 2013 - 2
IEEE Robotics & Automation Magazine - March 2013 - 3
IEEE Robotics & Automation Magazine - March 2013 - 4
IEEE Robotics & Automation Magazine - March 2013 - 5
IEEE Robotics & Automation Magazine - March 2013 - 6
IEEE Robotics & Automation Magazine - March 2013 - 7
IEEE Robotics & Automation Magazine - March 2013 - 8
IEEE Robotics & Automation Magazine - March 2013 - 9
IEEE Robotics & Automation Magazine - March 2013 - 10
IEEE Robotics & Automation Magazine - March 2013 - 11
IEEE Robotics & Automation Magazine - March 2013 - 12
IEEE Robotics & Automation Magazine - March 2013 - 13
IEEE Robotics & Automation Magazine - March 2013 - 14
IEEE Robotics & Automation Magazine - March 2013 - 15
IEEE Robotics & Automation Magazine - March 2013 - 16
IEEE Robotics & Automation Magazine - March 2013 - 17
IEEE Robotics & Automation Magazine - March 2013 - 18
IEEE Robotics & Automation Magazine - March 2013 - 19
IEEE Robotics & Automation Magazine - March 2013 - 20
IEEE Robotics & Automation Magazine - March 2013 - 21
IEEE Robotics & Automation Magazine - March 2013 - 22
IEEE Robotics & Automation Magazine - March 2013 - 23
IEEE Robotics & Automation Magazine - March 2013 - 24
IEEE Robotics & Automation Magazine - March 2013 - 25
IEEE Robotics & Automation Magazine - March 2013 - 26
IEEE Robotics & Automation Magazine - March 2013 - 27
IEEE Robotics & Automation Magazine - March 2013 - 28
IEEE Robotics & Automation Magazine - March 2013 - 29
IEEE Robotics & Automation Magazine - March 2013 - 30
IEEE Robotics & Automation Magazine - March 2013 - 31
IEEE Robotics & Automation Magazine - March 2013 - 32
IEEE Robotics & Automation Magazine - March 2013 - 33
IEEE Robotics & Automation Magazine - March 2013 - 34
IEEE Robotics & Automation Magazine - March 2013 - 35
IEEE Robotics & Automation Magazine - March 2013 - 36
IEEE Robotics & Automation Magazine - March 2013 - 37
IEEE Robotics & Automation Magazine - March 2013 - 38
IEEE Robotics & Automation Magazine - March 2013 - 39
IEEE Robotics & Automation Magazine - March 2013 - 40
IEEE Robotics & Automation Magazine - March 2013 - 41
IEEE Robotics & Automation Magazine - March 2013 - 42
IEEE Robotics & Automation Magazine - March 2013 - 43
IEEE Robotics & Automation Magazine - March 2013 - 44
IEEE Robotics & Automation Magazine - March 2013 - 45
IEEE Robotics & Automation Magazine - March 2013 - 46
IEEE Robotics & Automation Magazine - March 2013 - 47
IEEE Robotics & Automation Magazine - March 2013 - 48
IEEE Robotics & Automation Magazine - March 2013 - 49
IEEE Robotics & Automation Magazine - March 2013 - 50
IEEE Robotics & Automation Magazine - March 2013 - 51
IEEE Robotics & Automation Magazine - March 2013 - 52
IEEE Robotics & Automation Magazine - March 2013 - 53
IEEE Robotics & Automation Magazine - March 2013 - 54
IEEE Robotics & Automation Magazine - March 2013 - 55
IEEE Robotics & Automation Magazine - March 2013 - 56
IEEE Robotics & Automation Magazine - March 2013 - 57
IEEE Robotics & Automation Magazine - March 2013 - 58
IEEE Robotics & Automation Magazine - March 2013 - 59
IEEE Robotics & Automation Magazine - March 2013 - 60
IEEE Robotics & Automation Magazine - March 2013 - 61
IEEE Robotics & Automation Magazine - March 2013 - 62
IEEE Robotics & Automation Magazine - March 2013 - 63
IEEE Robotics & Automation Magazine - March 2013 - 64
IEEE Robotics & Automation Magazine - March 2013 - 65
IEEE Robotics & Automation Magazine - March 2013 - 66
IEEE Robotics & Automation Magazine - March 2013 - 67
IEEE Robotics & Automation Magazine - March 2013 - 68
IEEE Robotics & Automation Magazine - March 2013 - 69
IEEE Robotics & Automation Magazine - March 2013 - 70
IEEE Robotics & Automation Magazine - March 2013 - 71
IEEE Robotics & Automation Magazine - March 2013 - 72
IEEE Robotics & Automation Magazine - March 2013 - 73
IEEE Robotics & Automation Magazine - March 2013 - 74
IEEE Robotics & Automation Magazine - March 2013 - 75
IEEE Robotics & Automation Magazine - March 2013 - 76
IEEE Robotics & Automation Magazine - March 2013 - 77
IEEE Robotics & Automation Magazine - March 2013 - 78
IEEE Robotics & Automation Magazine - March 2013 - 79
IEEE Robotics & Automation Magazine - March 2013 - 80
IEEE Robotics & Automation Magazine - March 2013 - 81
IEEE Robotics & Automation Magazine - March 2013 - 82
IEEE Robotics & Automation Magazine - March 2013 - 83
IEEE Robotics & Automation Magazine - March 2013 - 84
IEEE Robotics & Automation Magazine - March 2013 - 85
IEEE Robotics & Automation Magazine - March 2013 - 86
IEEE Robotics & Automation Magazine - March 2013 - 87
IEEE Robotics & Automation Magazine - March 2013 - 88
IEEE Robotics & Automation Magazine - March 2013 - 89
IEEE Robotics & Automation Magazine - March 2013 - 90
IEEE Robotics & Automation Magazine - March 2013 - 91
IEEE Robotics & Automation Magazine - March 2013 - 92
IEEE Robotics & Automation Magazine - March 2013 - 93
IEEE Robotics & Automation Magazine - March 2013 - 94
IEEE Robotics & Automation Magazine - March 2013 - 95
IEEE Robotics & Automation Magazine - March 2013 - 96
IEEE Robotics & Automation Magazine - March 2013 - 97
IEEE Robotics & Automation Magazine - March 2013 - 98
IEEE Robotics & Automation Magazine - March 2013 - 99
IEEE Robotics & Automation Magazine - March 2013 - 100
IEEE Robotics & Automation Magazine - March 2013 - 101
IEEE Robotics & Automation Magazine - March 2013 - 102
IEEE Robotics & Automation Magazine - March 2013 - 103
IEEE Robotics & Automation Magazine - March 2013 - 104
IEEE Robotics & Automation Magazine - March 2013 - 105
IEEE Robotics & Automation Magazine - March 2013 - 106
IEEE Robotics & Automation Magazine - March 2013 - 107
IEEE Robotics & Automation Magazine - March 2013 - 108
IEEE Robotics & Automation Magazine - March 2013 - 109
IEEE Robotics & Automation Magazine - March 2013 - 110
IEEE Robotics & Automation Magazine - March 2013 - 111
IEEE Robotics & Automation Magazine - March 2013 - 112
IEEE Robotics & Automation Magazine - March 2013 - 113
IEEE Robotics & Automation Magazine - March 2013 - 114
IEEE Robotics & Automation Magazine - March 2013 - 115
IEEE Robotics & Automation Magazine - March 2013 - 116
IEEE Robotics & Automation Magazine - March 2013 - 117
IEEE Robotics & Automation Magazine - March 2013 - 118
IEEE Robotics & Automation Magazine - March 2013 - 119
IEEE Robotics & Automation Magazine - March 2013 - 120
IEEE Robotics & Automation Magazine - March 2013 - 121
IEEE Robotics & Automation Magazine - March 2013 - 122
IEEE Robotics & Automation Magazine - March 2013 - 123
IEEE Robotics & Automation Magazine - March 2013 - 124
IEEE Robotics & Automation Magazine - March 2013 - 125
IEEE Robotics & Automation Magazine - March 2013 - 126
IEEE Robotics & Automation Magazine - March 2013 - 127
IEEE Robotics & Automation Magazine - March 2013 - 128
IEEE Robotics & Automation Magazine - March 2013 - Cover3
IEEE Robotics & Automation Magazine - March 2013 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2010
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2010
https://www.nxtbookmedia.com