IEEE Technology and Society Magazine - September 2016 - 56

by characterizing authority-responsibility mismatches.
Authority describes which functions an agent is asked to
perform and responsibility describes which outcomes an
agent will be accountable for in an organizational, regulatory, or legal sense [30]. A synthesis of the function allocation literature makes a clear argument for a general
regulation of autonomous weapons systems: "the responsibility for the outcome of a function must be considered
relative to the authority to perform it" [33].
One well-studied mismatch comes from the modern
commercial airline cockpits where the human flight
crew "maintain responsibility for the safety of the flight,
even as the autopilot and autoflight systems exercise
significant authority over important actions within the
aircraft control and trajectory management functions"
[32]. In the interest of minimizing the number of mismatches, Pritchett et al. [32], [33] developed conceptual
models of various function allocations between the
human flight crew and the autopilot systems and counted the number of mismatches. Then, to understand the
effect of the various function allocations within different
contexts, the models were computationally simulated to
determine which one minimized human workload.
Those concerned about the AWS responsibility gaps
ought to use these methods developed in the cognitive
systems engineering literature to identify what functions
should be allocated to humans in order to establish the
desired authorities, responsibilities, and workload.

How Do We Review and Certify Permissible
Autonomous Weapons Systems?
At the 2016 U.N. CCW meeting on AWS, almost every
nation that addressed weapons reviews, endorsed the
idea that the review process was critical in ensuring the
lawful use of autonomous weapons [17]. In developing
such procedures for reviewing and certifying permissible
AWS, the experts should leverage the extensive modeling and measuring techniques for function allocation
that have been used extensively to analyze human-automation interaction.
There are two main categories of models for understanding the effectiveness of the distribution of work
between agents: static work models and dynamic simulations [32]. Static work models assess the distribution of
work in a context-independent manner, whereas dynamic
simulations focus on how the coupled interactions
between agents are affected by the context of the environment, usually through computational simulations.
Importantly these models should be developed through
significant analysis of the work domain via qualitative
and quantitative studies of the operations (most likely,
cognitive work analysis [48]).
We will focus our discussion on two static work models as they are more applicable to the AWS discussion at

56

this time, because we are not aware of any computational analyses of human-AWS interaction.4 One static work
model is the abstraction hierarchy, which provides an
understanding of how an agent may be able to view work
at different levels of abstraction, ranging from a detailed
descriptions of a specific task (manage decision to
engage) to mission goals (ensure safety of the convoy),
and modeling teamwork as defined by function allocation
and team design. A second model is to identify and list
the functions allocated to different agents within a team
(human operator and AWS) [32], a method we refer to as
function allocation form. In previous work [20] we used
function allocation form to identify and list the distribution of allocated functions between the human operator
and AWS, with the three classes of AWS as defined by the
DoD [28] and two definitions of MHC [22], [42]. Our analysis of the function allocation form showed that if either
MHC standard were adopted in the laws of armed conflict, all levels of the DoD's AWS would be illegal because
of one specific difference in the definitions: whether the
human (required by MHC) or the AWS (as classified by
DoD) has control over engaging targets.
Once the models are completed, they can be evaluated
using at least eight different metrics [33]: 1) workload, 2)
stability of the work environment, 3) mismatches between
responsibility and authority, 4) incoherency in function
allocations, 5) interruptive automation, 6) automation's
boundary conditions, 7) function allocations limiting
human adaptation to context, 8) and mission performance.
Theoretically, it would be possible to evaluate the currently proposed definitions of AWS and MHC using these metrics to determine how the function allocations actually
relate to the performance of the human-AWS team.

The Importance of a Technical Language
Our argument for using a technical language for discussing AWS is not new; nor is it without detractors. When
discussing the definitions of AWS, Scharre and Horowitz
[43] argued that autonomy should be described along
three dimensions: the human-machine command-andcontrol relationship; the complexity of the machine; and
the type of decision being automated. They concluded
their descriptions by stating, "These are all important
features of autonomous systems, but they are different
ideas, and often people tend to mix them together."
Crootof [16] rebutted these three dimensions of
autonomy, stating that, "evaluating autonomy along
these different axes introduces a confusing and unnecessary level of particularity... Although these gradations
may be extremely useful in research and development,
they are unnecessarily precise for a legal document that
4

There are computational analyses of function allocation in other complex
work domains such as pilot-autopilot interaction in commercial aviation
[37], [38].

IEEE Technology and Society Magazine

∕

september 2016



Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2016

IEEE Technology and Society Magazine - September 2016 - Cover1
IEEE Technology and Society Magazine - September 2016 - Cover2
IEEE Technology and Society Magazine - September 2016 - 1
IEEE Technology and Society Magazine - September 2016 - 2
IEEE Technology and Society Magazine - September 2016 - 3
IEEE Technology and Society Magazine - September 2016 - 4
IEEE Technology and Society Magazine - September 2016 - 5
IEEE Technology and Society Magazine - September 2016 - 6
IEEE Technology and Society Magazine - September 2016 - 7
IEEE Technology and Society Magazine - September 2016 - 8
IEEE Technology and Society Magazine - September 2016 - 9
IEEE Technology and Society Magazine - September 2016 - 10
IEEE Technology and Society Magazine - September 2016 - 11
IEEE Technology and Society Magazine - September 2016 - 12
IEEE Technology and Society Magazine - September 2016 - 13
IEEE Technology and Society Magazine - September 2016 - 14
IEEE Technology and Society Magazine - September 2016 - 15
IEEE Technology and Society Magazine - September 2016 - 16
IEEE Technology and Society Magazine - September 2016 - 17
IEEE Technology and Society Magazine - September 2016 - 18
IEEE Technology and Society Magazine - September 2016 - 19
IEEE Technology and Society Magazine - September 2016 - 20
IEEE Technology and Society Magazine - September 2016 - 21
IEEE Technology and Society Magazine - September 2016 - 22
IEEE Technology and Society Magazine - September 2016 - 23
IEEE Technology and Society Magazine - September 2016 - 24
IEEE Technology and Society Magazine - September 2016 - 25
IEEE Technology and Society Magazine - September 2016 - 26
IEEE Technology and Society Magazine - September 2016 - 27
IEEE Technology and Society Magazine - September 2016 - 28
IEEE Technology and Society Magazine - September 2016 - 29
IEEE Technology and Society Magazine - September 2016 - 30
IEEE Technology and Society Magazine - September 2016 - 31
IEEE Technology and Society Magazine - September 2016 - 32
IEEE Technology and Society Magazine - September 2016 - 33
IEEE Technology and Society Magazine - September 2016 - 34
IEEE Technology and Society Magazine - September 2016 - 35
IEEE Technology and Society Magazine - September 2016 - 36
IEEE Technology and Society Magazine - September 2016 - 37
IEEE Technology and Society Magazine - September 2016 - 38
IEEE Technology and Society Magazine - September 2016 - 39
IEEE Technology and Society Magazine - September 2016 - 40
IEEE Technology and Society Magazine - September 2016 - 41
IEEE Technology and Society Magazine - September 2016 - 42
IEEE Technology and Society Magazine - September 2016 - 43
IEEE Technology and Society Magazine - September 2016 - 44
IEEE Technology and Society Magazine - September 2016 - 45
IEEE Technology and Society Magazine - September 2016 - 46
IEEE Technology and Society Magazine - September 2016 - 47
IEEE Technology and Society Magazine - September 2016 - 48
IEEE Technology and Society Magazine - September 2016 - 49
IEEE Technology and Society Magazine - September 2016 - 50
IEEE Technology and Society Magazine - September 2016 - 51
IEEE Technology and Society Magazine - September 2016 - 52
IEEE Technology and Society Magazine - September 2016 - 53
IEEE Technology and Society Magazine - September 2016 - 54
IEEE Technology and Society Magazine - September 2016 - 55
IEEE Technology and Society Magazine - September 2016 - 56
IEEE Technology and Society Magazine - September 2016 - 57
IEEE Technology and Society Magazine - September 2016 - 58
IEEE Technology and Society Magazine - September 2016 - 59
IEEE Technology and Society Magazine - September 2016 - 60
IEEE Technology and Society Magazine - September 2016 - 61
IEEE Technology and Society Magazine - September 2016 - 62
IEEE Technology and Society Magazine - September 2016 - 63
IEEE Technology and Society Magazine - September 2016 - 64
IEEE Technology and Society Magazine - September 2016 - 65
IEEE Technology and Society Magazine - September 2016 - 66
IEEE Technology and Society Magazine - September 2016 - 67
IEEE Technology and Society Magazine - September 2016 - 68
IEEE Technology and Society Magazine - September 2016 - 69
IEEE Technology and Society Magazine - September 2016 - 70
IEEE Technology and Society Magazine - September 2016 - 71
IEEE Technology and Society Magazine - September 2016 - 72
IEEE Technology and Society Magazine - September 2016 - 73
IEEE Technology and Society Magazine - September 2016 - 74
IEEE Technology and Society Magazine - September 2016 - 75
IEEE Technology and Society Magazine - September 2016 - 76
IEEE Technology and Society Magazine - September 2016 - 77
IEEE Technology and Society Magazine - September 2016 - 78
IEEE Technology and Society Magazine - September 2016 - 79
IEEE Technology and Society Magazine - September 2016 - 80
IEEE Technology and Society Magazine - September 2016 - 81
IEEE Technology and Society Magazine - September 2016 - 82
IEEE Technology and Society Magazine - September 2016 - 83
IEEE Technology and Society Magazine - September 2016 - 84
IEEE Technology and Society Magazine - September 2016 - 85
IEEE Technology and Society Magazine - September 2016 - 86
IEEE Technology and Society Magazine - September 2016 - 87
IEEE Technology and Society Magazine - September 2016 - 88
IEEE Technology and Society Magazine - September 2016 - Cover3
IEEE Technology and Society Magazine - September 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2013
https://www.nxtbookmedia.com