IEEE Technology and Society Magazine - September 2016 - 57
is ultimately concerned with regulating weaponry that
might independently exercise lethal force."
While more work should be done to address any confusing aspects of technical explanations of AWS, we argue
that particularity and precision are absolutely necessary
for understanding and regulating AWS. Just a cursory
review of the incidents and accidents related to humanautomation team problems shows that even well-intentioned designs cause serious harm [34], [38], [44].
Cognitive systems engineering and its related disciplines
have spent decades trying to unmask the causes of the
harm and, as shown in this paper, developed some
important techniques for designing effective teams such
that harm is mitigated. This is why the fifth requirement
for effective function allocation explicitly states that function allocation (or cognitive systems engineering, generally) should be a part of the design process. Therefore, if
ethicists and regulators want designers to address meaningful human control or responsibility gaps in their
designs, then the regulations will eventually need to be
translated into particular and precise technical language
like cognitive systems engineering.
Another benefit of adopting technical languages is
that they are often agnostic to the ethical challenges
being debated. In a "pro-regulation world," where a fully
AWS is legal, regulating its design according to the
requirements of function allocation would still work.
This is because the AWS would be required to perform
its functions and support the dynamics of the work -
including upholding the laws of war. Alternatively, in a
"pro-ban world" the arguments of requiring meaningful
human control can be integrated into the requirements.
For example, should the U.N. adopt a proposal of meaningful human control such that only human operators
can perform the function of engaging targets, then the
functions should be allocated to the human and AWS
such that both can perform their functions with reasonable teamwork within the dynamics of the work.
More broadly, if the regulators do not understand the
real complexities of the systems they are trying to regulate
then it is easy for those who are constructing and fielding
the technology to, either deliberately or unintentionally, discover loopholes in the regulations. This concern has been
termed the "pacing problem" to reflect the growing gap
between the pace of science and technology and the lagging responsiveness of the legal and ethical oversight that
society relies on to govern emerging technologies [45]. This
pacing problem is not unique to AWS as it affects the areas
of biotechnology, genetic testing, nanotechnology, computer privacy, and many others. However, for AWS in particular,
our legal and ethical discussions cannot shirk the complexities of the human-AWS teams because that could result in
insufficient regulations potentially enabling the use of fully
AWS under the guise of meaningful human control.
september 2016
∕
Cognitive Systems Engineering: The Language
of Autonomous Weapons Systems Regulations
The language of cognitive systems engineering, its principles, models, and measures, have been developed over
decades to address concerns associated with designing,
analyzing, and evaluating human-autonomous teams.
Our motivation for showing how the cognitive systems
engineering language can address the four key issues of
U.N. CCW discussions of AWS (defining autonomy,
human control, accountability, and review and certification) is our concern that the international community
may pass AWS regulations that do not result in the
intended consequences. If an insufficient definition of
MHC is adopted, AWS could be fielded that pass the regulatory tests, but that only have a façade of human control
determining life or death - leaving the human operator
responsible for actions they cannot be expected to control. Furthermore, if an AWS is incorrectly classified and
insufficiently regulated, the determinants of responsibility for the actions of the AWS will be too ambiguous - fulfilling the concern that operators, programmers, or
manufacturers who should be found responsible will
escape liability.
We intend for this article to differentially affect two
sets of communities. First, we hope it will provide the
experts and researchers that are already participants in
the AWS debates and discussions with new language for
articulating their ethical concerns and proposals for regulations. Using cognitive systems engineering as a common language should enable identification of the
sources of conflict and solutions that have consensus,
and operationalize regulations that are agreed upon.
Second, we hope that cognitive systems engineers5 will
see the value of their perspectives and expertise to the
ethical and legal communities, where regulating humanautomation teams are becoming both more important
and more difficult - and thus start to engage in the
debates and discussions themselves.
There are many avenues for future work at the intersection of technology regulation and cognitive systems
engineering. For the specific problem of human-AWS
teams we intend to identify what function allocations are
necessary for a human-AWS team if the direct human operator is responsible for upholding the laws of war. Moreover, the synergy between cognitive systems engineering
and the AWS discussions are only an example of the synergies between cognitive systems engineering and the
regulation of robotics and automated systems generally.
We believe there is potential for the formal integration of
cognitive systems engineering as a partner to the growing
field of "robot law" [46].
5
This includes all those related to cognitive systems engineering regardless of their labeled discipline: psychologists, engineers, designers, and
scientists, etc.
IEEE Technology and Society Magazine
57
Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - September 2016
IEEE Technology and Society Magazine - September 2016 - Cover1
IEEE Technology and Society Magazine - September 2016 - Cover2
IEEE Technology and Society Magazine - September 2016 - 1
IEEE Technology and Society Magazine - September 2016 - 2
IEEE Technology and Society Magazine - September 2016 - 3
IEEE Technology and Society Magazine - September 2016 - 4
IEEE Technology and Society Magazine - September 2016 - 5
IEEE Technology and Society Magazine - September 2016 - 6
IEEE Technology and Society Magazine - September 2016 - 7
IEEE Technology and Society Magazine - September 2016 - 8
IEEE Technology and Society Magazine - September 2016 - 9
IEEE Technology and Society Magazine - September 2016 - 10
IEEE Technology and Society Magazine - September 2016 - 11
IEEE Technology and Society Magazine - September 2016 - 12
IEEE Technology and Society Magazine - September 2016 - 13
IEEE Technology and Society Magazine - September 2016 - 14
IEEE Technology and Society Magazine - September 2016 - 15
IEEE Technology and Society Magazine - September 2016 - 16
IEEE Technology and Society Magazine - September 2016 - 17
IEEE Technology and Society Magazine - September 2016 - 18
IEEE Technology and Society Magazine - September 2016 - 19
IEEE Technology and Society Magazine - September 2016 - 20
IEEE Technology and Society Magazine - September 2016 - 21
IEEE Technology and Society Magazine - September 2016 - 22
IEEE Technology and Society Magazine - September 2016 - 23
IEEE Technology and Society Magazine - September 2016 - 24
IEEE Technology and Society Magazine - September 2016 - 25
IEEE Technology and Society Magazine - September 2016 - 26
IEEE Technology and Society Magazine - September 2016 - 27
IEEE Technology and Society Magazine - September 2016 - 28
IEEE Technology and Society Magazine - September 2016 - 29
IEEE Technology and Society Magazine - September 2016 - 30
IEEE Technology and Society Magazine - September 2016 - 31
IEEE Technology and Society Magazine - September 2016 - 32
IEEE Technology and Society Magazine - September 2016 - 33
IEEE Technology and Society Magazine - September 2016 - 34
IEEE Technology and Society Magazine - September 2016 - 35
IEEE Technology and Society Magazine - September 2016 - 36
IEEE Technology and Society Magazine - September 2016 - 37
IEEE Technology and Society Magazine - September 2016 - 38
IEEE Technology and Society Magazine - September 2016 - 39
IEEE Technology and Society Magazine - September 2016 - 40
IEEE Technology and Society Magazine - September 2016 - 41
IEEE Technology and Society Magazine - September 2016 - 42
IEEE Technology and Society Magazine - September 2016 - 43
IEEE Technology and Society Magazine - September 2016 - 44
IEEE Technology and Society Magazine - September 2016 - 45
IEEE Technology and Society Magazine - September 2016 - 46
IEEE Technology and Society Magazine - September 2016 - 47
IEEE Technology and Society Magazine - September 2016 - 48
IEEE Technology and Society Magazine - September 2016 - 49
IEEE Technology and Society Magazine - September 2016 - 50
IEEE Technology and Society Magazine - September 2016 - 51
IEEE Technology and Society Magazine - September 2016 - 52
IEEE Technology and Society Magazine - September 2016 - 53
IEEE Technology and Society Magazine - September 2016 - 54
IEEE Technology and Society Magazine - September 2016 - 55
IEEE Technology and Society Magazine - September 2016 - 56
IEEE Technology and Society Magazine - September 2016 - 57
IEEE Technology and Society Magazine - September 2016 - 58
IEEE Technology and Society Magazine - September 2016 - 59
IEEE Technology and Society Magazine - September 2016 - 60
IEEE Technology and Society Magazine - September 2016 - 61
IEEE Technology and Society Magazine - September 2016 - 62
IEEE Technology and Society Magazine - September 2016 - 63
IEEE Technology and Society Magazine - September 2016 - 64
IEEE Technology and Society Magazine - September 2016 - 65
IEEE Technology and Society Magazine - September 2016 - 66
IEEE Technology and Society Magazine - September 2016 - 67
IEEE Technology and Society Magazine - September 2016 - 68
IEEE Technology and Society Magazine - September 2016 - 69
IEEE Technology and Society Magazine - September 2016 - 70
IEEE Technology and Society Magazine - September 2016 - 71
IEEE Technology and Society Magazine - September 2016 - 72
IEEE Technology and Society Magazine - September 2016 - 73
IEEE Technology and Society Magazine - September 2016 - 74
IEEE Technology and Society Magazine - September 2016 - 75
IEEE Technology and Society Magazine - September 2016 - 76
IEEE Technology and Society Magazine - September 2016 - 77
IEEE Technology and Society Magazine - September 2016 - 78
IEEE Technology and Society Magazine - September 2016 - 79
IEEE Technology and Society Magazine - September 2016 - 80
IEEE Technology and Society Magazine - September 2016 - 81
IEEE Technology and Society Magazine - September 2016 - 82
IEEE Technology and Society Magazine - September 2016 - 83
IEEE Technology and Society Magazine - September 2016 - 84
IEEE Technology and Society Magazine - September 2016 - 85
IEEE Technology and Society Magazine - September 2016 - 86
IEEE Technology and Society Magazine - September 2016 - 87
IEEE Technology and Society Magazine - September 2016 - 88
IEEE Technology and Society Magazine - September 2016 - Cover3
IEEE Technology and Society Magazine - September 2016 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2023
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2022
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2021
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2020
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2019
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_december2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_september2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_june2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_march2018
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2017
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2016
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2015
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2014
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_winter2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_fall2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_summer2013
https://www.nxtbook.com/nxtbooks/ieee/technologysociety_spring2013
https://www.nxtbookmedia.com