IEEE Robotics & Automation Magazine - December 2015 - 141

can be quantified by the mean time (MT) to complete a
task and the number of collisions between the manipulator
and the environment. It has been postulated that the
speed-accuracy tradeoff stems from a signal-dependent
neuromotor noise. That is, faster movements require greater forces in the muscles, which again introduces more neuromotor noise and movement variability [17]. An increase
in the signal-dependent neuromotor noise has been related
to stroke-related motor deficiencies [18].
It is clear that real end-user participants are needed to validate the clinical credibility of any assistive technology. However, the involvement of people with disabilities in a lengthy
and tiring systematic activity of validation in intermediate
stages of development must be reasonably limited. It can also
be challenging to find a homogeneous set of disabled participants. Experimental comparisons can be made more reliable
and easier to replicate by simulating a consistent disability for
a set of able-bodied participants. That is, by introducing controlled perturbations in the perception-action loop. This
shortens the prototype development and facilitates the experimentation on more robotics-specific research issues. However, the simulation of disabilities in able-bodied people is
not obvious, as the neurophysiological modeling of disabilities is still a research issue. When a specific model cannot be
obtained, or when it is not appropriate to focus on only one
disability, a complementary approach is to assume a simplified physical disability that reduces the ability of the operator
to control the system. Previous examples of this approach include assessments of computer mouse movements by disabled people [19] and of shared control with joystick input
for assistive wheelchairs [8].
For the experiments reported here, an increase in the signal-dependent noise was used as a first-order approximation
of a physical disability. A Gaussian noise (Z in Figure 3) was
added to the raw able-bodied user input, in analogy with
[19]. The noise was filtered to below 2 Hz to be comparable
with typical human movements in the frequency domain. It
was also made to increase in strength with the magnitude of
the velocities commanded by the user, a signal-dependent
noise making the speed-accuracy tradeoff more difficult
[16]-[18]. Note that a disability can also require the use of
less dexterous parts of the body for robot control, which are
also typically subject to the speed-accuracy tradeoff. While
the noise added does not necessarily correspond exactly to a
specific real disability, it is an attempt at emulating the negative effect a disability could have on the ability of the user to
accurately control the manipulator.
Performance Metrics
Given the previously mentioned models of system and user,
what metrics are suitable for measuring performance? The
MT to completion is a natural metric when measuring performance on tasks involving movements. But, given the
speed-accuracy tradeoff, the high speeds associated with low
completion times will typically lower the accuracy and, thus,
increase errors. In some experiments on simple movements

(e.g., the Fitts' law paradigm), the participants are instructed
to maintain the error rate below a certain level (typically
2-5%). This helps disambiguate results with low times but a
high number of errors, and vice versa, but is difficult to control for more complex tasks. Here, we include the consequences of an error in the MT by making each collision cost
time. Put in the context of the signal-dependent noise in the
human sensorimotor system [17], the user has to adjust the
speed of execution to achieve a variance of the trajectories
performed (variance of X in Figure 3) that statistically minimizes the average MT over attempts. This is the main performance metric used here, but it says little about the inner
workings of the human-robot system.
Controllability is a useful metric for controllers acting in
a closed loop in general. Loosely speaking, a system is controllable if it can be commanded to any final state from any
initial state. For complex stochastic systems, a different formulation with broader conditions is desirable, such as the
information-theoretic approach presented in [12]. Following this approach, we showed that controllability is also relevant for the specific case where the controller is a disabled
user, H, trying to overcome the noise produced by his/her
disabilities, Z, to successfully perform a manipulation task
with a physically assistive robot [4]. For the experimental
evaluation performed here, the controllability is approximated using the unconditional mutual information over
the robot actuation and the noise added ( A t and Z t in Figure 3). That is, we used I ^ A t; Z t h, measured over a finite set
of tasks and a fixed number of repetitions for each task. The
controllability metric, thus, measures the correlation of the
noise added with the actuation of the robot. A lower correlation indicates that the system is better at rejecting the
noise and has a higher controllability from the user's perspective. The simplification assumes that: 1) the user attempts movements that are repeatable (i.e., passing through
similar trajectories of X t and H t ) and 2) the movements
are close to optimal.
Enforcing such conditions is not always possible, however. A random user input could, for example, artificially inflate the controllability approximation used. Bialek et al. [20]
proposed PI, the mutual information over the past and the
future, as a general measure of the complexity of a time series. The measure can be said to quantify the total information of past experience that can be used for predicting future
events, and has, among others, been used to drive the behavior of mobile robots in unknown environments. Applied
to the robot actuation in Figure 3, it may be used to compare how predictable the actuation is with different sharedcontrol approaches. The one-step mutual information was
here used, with the PI of actuation thus being defined as
I ^ A t; A t +1h . The measure can also be used to identify cases
where the controllability metric is no longer valid.
Case Study 1
Two case studies are used here as examples for the approach
to replication outlined in the previous section. Both case
DECEMBER 2015

IEEE ROBOTICS & AUTOMATION MAGAZINE

141

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2015