IEEE Robotics & Automation Magazine - March 2013 - 56
amount of time to return an outcome. Anticipations of
this kind represent a form of knowledge that is common
but falls outside the learning capabilities of most standard pattern recognition approaches. To date, there are
few approaches able to learn this form of anticipatory
knowledge for real-valued signals, and fewer still can
learn and continually update (adapt) this type of predictive representation during online, real-time operation.
RL is one form of machine learning that has demonstrated the ability to learn in an ongoing, incremental prediction and control setting [14]. An RL system uses
interactions with its environment to build up expectations
about future events. Specifically, it learns to estimate the
value of a one-dimensional (1-D) feedback signal termed
reward; these estimates are often represented using a value
function (a mapping from observations of the environment to expectations about future reward).
RL is viewed as an approach to artificial intelligence,
natural intelligence, optimal control, and operations
research. Since their development in the 1980s, RL algorithms have been widely used in robotics and have found
the best-known approximate solutions to many games;
they have also become the standard model of reward processing in the brain [15].
Recent work has provided a straightforward way to
use RL for acquiring expectations and value functions
pertaining to nonreward signals and observations [16].
These general value functions (GVFs) are proposed as a
way of asking and answering temporally extended questions about future sensorimotor experience. Predictive
questions can be defined for different time scales and
may take into account different methods for weighting
the importance of future observations. The anticipations
learned using GVFs can also depend on numerous strategies for choosing control actions (policies) and can be
defined for events with no fixed length [16]. Expectations comprising a GVF are acquired using standard RL
techniques; this means that learning can occur in an
incremental, online fashion, with constant demands in
terms of both memory and computation. The approach
we develop in this article is to apply GVFs alongside
myoelectric control.
Formalizing Predictions with GVFs
We use the standard RL framework of states (s ! S) ,
actions (a ! A), time-steps t $ 0, and rewards (r ! R)
[14]. In our context, a GVF represents a question q about a
scalar signal of interest, here denoted r for consistency; this
question depends on a given probability function for
choosing actions r : S # A " [0, 1] and a temporal continuation probability c : S " [0, 1] . A question q may therefore be written as: "given state s, what is the expected value
of the cumulative sum of a signal r while following a policy
r, and while continuing with a probability given by c? "
Formally, the value function Vq (s) for our question is
defined as follows, where actions are taken according to r
56
*
IEEE ROBOTICS & AUTOMATION MAGAZINE
*
March 2013
and there exist state-dependent continuation probabilities
0 # c (s) # 1 for all s ! S
3
k
k =0
i =1
Vq (s) = E r = / e % c (s t +i) o r (s t +k +1) ; s t = sG .
This defines the exact answer to a question when states
are fully observable. In practice, a state is rarely fully
observable or needs to be approximated to represent an
answer to a question. We instead assume that we observe a
vector of features x that depends on the current state s
according to some state approximation function x !
approx (s). We can then present the approximate answer
Vt q (s) to a question q as a prediction Pq that is the linear
combination of a (learned) weight vector w and the feature
vector x at time t
Pq = Vt q (s) = w q< x.
For our research, the approximation function approx(s)
was implemented using tile coding, as per Sutton and Barto
[14]. Tile coding is a linear mathematical function that maps
a real-valued signal space into a linear (binary) vector form
that can be used for efficient computation and learning [14].
To facilitate incremental computation, in what follows we
consider the exponentially discounted case of GVFs, where
0 # c # 1, and c is the same for all states in the system. The
value for state s is therefore the expected sum of an exponentially discounted signal r for each future time step
Vq (s) = E r ; /
3
c
k =0
k
r (s t +k +1) ; s t = sE .
The cumulative value inside this expectation is termed the
return. For the purposes of post hoc comparison, the true
return R q on a time-step t may be computed by recording
future experience over a window of T data points, where T is
P1
P2
GVF 1
r1
P3
GVF 2
r2
PN
GVF 3
GVF N
r3
rN
State (x)
Approx
FXN APP
Control
Human
Robot
EMG
Signals
Actuator
Signals
Feedback Signals
Control Signals
Figure 2. A schematic showing how GVFs predict the expected future
value of signals from the sensorimotor space of a myoelectric control
system. Each GVF learns temporally extended predictions pq about
a specific signal of interest rq. Predictions are learned with respect to
the current state of the system, as represented by the feature vector x.
This feature vector is generated from the observed sensorimotor signal
space using a function approximation routine, shown here as FXN APP.
Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - March 2013
IEEE Robotics & Automation Magazine - March 2013 - Cover1
IEEE Robotics & Automation Magazine - March 2013 - Cover2
IEEE Robotics & Automation Magazine - March 2013 - 1
IEEE Robotics & Automation Magazine - March 2013 - 2
IEEE Robotics & Automation Magazine - March 2013 - 3
IEEE Robotics & Automation Magazine - March 2013 - 4
IEEE Robotics & Automation Magazine - March 2013 - 5
IEEE Robotics & Automation Magazine - March 2013 - 6
IEEE Robotics & Automation Magazine - March 2013 - 7
IEEE Robotics & Automation Magazine - March 2013 - 8
IEEE Robotics & Automation Magazine - March 2013 - 9
IEEE Robotics & Automation Magazine - March 2013 - 10
IEEE Robotics & Automation Magazine - March 2013 - 11
IEEE Robotics & Automation Magazine - March 2013 - 12
IEEE Robotics & Automation Magazine - March 2013 - 13
IEEE Robotics & Automation Magazine - March 2013 - 14
IEEE Robotics & Automation Magazine - March 2013 - 15
IEEE Robotics & Automation Magazine - March 2013 - 16
IEEE Robotics & Automation Magazine - March 2013 - 17
IEEE Robotics & Automation Magazine - March 2013 - 18
IEEE Robotics & Automation Magazine - March 2013 - 19
IEEE Robotics & Automation Magazine - March 2013 - 20
IEEE Robotics & Automation Magazine - March 2013 - 21
IEEE Robotics & Automation Magazine - March 2013 - 22
IEEE Robotics & Automation Magazine - March 2013 - 23
IEEE Robotics & Automation Magazine - March 2013 - 24
IEEE Robotics & Automation Magazine - March 2013 - 25
IEEE Robotics & Automation Magazine - March 2013 - 26
IEEE Robotics & Automation Magazine - March 2013 - 27
IEEE Robotics & Automation Magazine - March 2013 - 28
IEEE Robotics & Automation Magazine - March 2013 - 29
IEEE Robotics & Automation Magazine - March 2013 - 30
IEEE Robotics & Automation Magazine - March 2013 - 31
IEEE Robotics & Automation Magazine - March 2013 - 32
IEEE Robotics & Automation Magazine - March 2013 - 33
IEEE Robotics & Automation Magazine - March 2013 - 34
IEEE Robotics & Automation Magazine - March 2013 - 35
IEEE Robotics & Automation Magazine - March 2013 - 36
IEEE Robotics & Automation Magazine - March 2013 - 37
IEEE Robotics & Automation Magazine - March 2013 - 38
IEEE Robotics & Automation Magazine - March 2013 - 39
IEEE Robotics & Automation Magazine - March 2013 - 40
IEEE Robotics & Automation Magazine - March 2013 - 41
IEEE Robotics & Automation Magazine - March 2013 - 42
IEEE Robotics & Automation Magazine - March 2013 - 43
IEEE Robotics & Automation Magazine - March 2013 - 44
IEEE Robotics & Automation Magazine - March 2013 - 45
IEEE Robotics & Automation Magazine - March 2013 - 46
IEEE Robotics & Automation Magazine - March 2013 - 47
IEEE Robotics & Automation Magazine - March 2013 - 48
IEEE Robotics & Automation Magazine - March 2013 - 49
IEEE Robotics & Automation Magazine - March 2013 - 50
IEEE Robotics & Automation Magazine - March 2013 - 51
IEEE Robotics & Automation Magazine - March 2013 - 52
IEEE Robotics & Automation Magazine - March 2013 - 53
IEEE Robotics & Automation Magazine - March 2013 - 54
IEEE Robotics & Automation Magazine - March 2013 - 55
IEEE Robotics & Automation Magazine - March 2013 - 56
IEEE Robotics & Automation Magazine - March 2013 - 57
IEEE Robotics & Automation Magazine - March 2013 - 58
IEEE Robotics & Automation Magazine - March 2013 - 59
IEEE Robotics & Automation Magazine - March 2013 - 60
IEEE Robotics & Automation Magazine - March 2013 - 61
IEEE Robotics & Automation Magazine - March 2013 - 62
IEEE Robotics & Automation Magazine - March 2013 - 63
IEEE Robotics & Automation Magazine - March 2013 - 64
IEEE Robotics & Automation Magazine - March 2013 - 65
IEEE Robotics & Automation Magazine - March 2013 - 66
IEEE Robotics & Automation Magazine - March 2013 - 67
IEEE Robotics & Automation Magazine - March 2013 - 68
IEEE Robotics & Automation Magazine - March 2013 - 69
IEEE Robotics & Automation Magazine - March 2013 - 70
IEEE Robotics & Automation Magazine - March 2013 - 71
IEEE Robotics & Automation Magazine - March 2013 - 72
IEEE Robotics & Automation Magazine - March 2013 - 73
IEEE Robotics & Automation Magazine - March 2013 - 74
IEEE Robotics & Automation Magazine - March 2013 - 75
IEEE Robotics & Automation Magazine - March 2013 - 76
IEEE Robotics & Automation Magazine - March 2013 - 77
IEEE Robotics & Automation Magazine - March 2013 - 78
IEEE Robotics & Automation Magazine - March 2013 - 79
IEEE Robotics & Automation Magazine - March 2013 - 80
IEEE Robotics & Automation Magazine - March 2013 - 81
IEEE Robotics & Automation Magazine - March 2013 - 82
IEEE Robotics & Automation Magazine - March 2013 - 83
IEEE Robotics & Automation Magazine - March 2013 - 84
IEEE Robotics & Automation Magazine - March 2013 - 85
IEEE Robotics & Automation Magazine - March 2013 - 86
IEEE Robotics & Automation Magazine - March 2013 - 87
IEEE Robotics & Automation Magazine - March 2013 - 88
IEEE Robotics & Automation Magazine - March 2013 - 89
IEEE Robotics & Automation Magazine - March 2013 - 90
IEEE Robotics & Automation Magazine - March 2013 - 91
IEEE Robotics & Automation Magazine - March 2013 - 92
IEEE Robotics & Automation Magazine - March 2013 - 93
IEEE Robotics & Automation Magazine - March 2013 - 94
IEEE Robotics & Automation Magazine - March 2013 - 95
IEEE Robotics & Automation Magazine - March 2013 - 96
IEEE Robotics & Automation Magazine - March 2013 - 97
IEEE Robotics & Automation Magazine - March 2013 - 98
IEEE Robotics & Automation Magazine - March 2013 - 99
IEEE Robotics & Automation Magazine - March 2013 - 100
IEEE Robotics & Automation Magazine - March 2013 - 101
IEEE Robotics & Automation Magazine - March 2013 - 102
IEEE Robotics & Automation Magazine - March 2013 - 103
IEEE Robotics & Automation Magazine - March 2013 - 104
IEEE Robotics & Automation Magazine - March 2013 - 105
IEEE Robotics & Automation Magazine - March 2013 - 106
IEEE Robotics & Automation Magazine - March 2013 - 107
IEEE Robotics & Automation Magazine - March 2013 - 108
IEEE Robotics & Automation Magazine - March 2013 - 109
IEEE Robotics & Automation Magazine - March 2013 - 110
IEEE Robotics & Automation Magazine - March 2013 - 111
IEEE Robotics & Automation Magazine - March 2013 - 112
IEEE Robotics & Automation Magazine - March 2013 - 113
IEEE Robotics & Automation Magazine - March 2013 - 114
IEEE Robotics & Automation Magazine - March 2013 - 115
IEEE Robotics & Automation Magazine - March 2013 - 116
IEEE Robotics & Automation Magazine - March 2013 - 117
IEEE Robotics & Automation Magazine - March 2013 - 118
IEEE Robotics & Automation Magazine - March 2013 - 119
IEEE Robotics & Automation Magazine - March 2013 - 120
IEEE Robotics & Automation Magazine - March 2013 - 121
IEEE Robotics & Automation Magazine - March 2013 - 122
IEEE Robotics & Automation Magazine - March 2013 - 123
IEEE Robotics & Automation Magazine - March 2013 - 124
IEEE Robotics & Automation Magazine - March 2013 - 125
IEEE Robotics & Automation Magazine - March 2013 - 126
IEEE Robotics & Automation Magazine - March 2013 - 127
IEEE Robotics & Automation Magazine - March 2013 - 128
IEEE Robotics & Automation Magazine - March 2013 - Cover3
IEEE Robotics & Automation Magazine - March 2013 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2023
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2022
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2021
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2020
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2019
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2018
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2017
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2016
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2015
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2014
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2013
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2012
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_june2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_march2011
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_december2010
https://www.nxtbook.com/nxtbooks/ieee/roboticsautomation_september2010
https://www.nxtbookmedia.com