IEEE Technology and Society Magazine - June 2019 - 52

not be hard to predict what police might do with a "heat
list," if one has a realistic sense of police work and the
pressures operating within precincts and departments.
This again points to the need for domain experts and
participatory design [56]. One imagines that a police
sergeant on the design team of the SSL would have
pointed out the likely misuses of the system. Prototyping and testing could also help reveal such tendencies,
as well as short-term and long-term evaluations of the
system implementation.
Transparency over the algorithms, data, and practices of implementation are also necessary. While the Chicago Police Department sought to avoid embarrassment
from releasing the details of the SSL, it would be impossible for independent outside researchers to evaluate its
impacts - positive and negative - without access to
the data and algorithms. It should not take a prolonged
lawsuit from a newspaper for government agencies to
share public data. Of course, as more and more commercial systems, like PredPol,19 make the algorithms
and even the data proprietary, they will fall under intellectual property protections. This means private companies will be processing the data, and will not be
required to reveal their algorithms, or subject them to
independent outside scrutiny. In some cases, private
companies are even withholding crime data from the
cities that produced it because they have formatted it in
a database for their system and even encrypted it such
that it cannot be used if the city changes to another
software platform [59].

Central Issues Facing AI Predictive Policing
It is hoped that this article has shed light upon some of
the central issues facing AI ethics in general and predictive policing in particular. While the use of data and AI in
policing is not intrinsically or necessarily unethical, it
must be done with care to avoid unjust and unethical
impacts. First among these issues is that while AI ethics
needs to understand the computational techniques it
deploys, it also needs a critical understanding of the
datasets it operates on, how data is collected, and the
social organizations and the biases that those datasets
may represent. This requires understanding how data
practices are embedded within socio-technical systems,
and not blindly analyzing data assuming that it is without
bias. It is also important to understand how the use of AI
tools and techniques will impact the beliefs and practices of those who engage with them. Datasets and their
computational analysis have the power to "makeup people" in the sense of Hacking [36], and also to prejudge
them according to statistical patterns and categories.
19

PredPol is a commercial software company developing data management
and predictive data systems for police departments [30].

Even when statistically justified, such categories, and the
actions of government agents on the basis of those categories, may disrespect individual rights, human dignity,
and undermine justice.
By taking an Ethics of Care approach to AI systems
design and ethics, designers should have a greater awareness and respect for these issues. While any design
approach is ultimately limited in its ability to mitigate all
possible failures and harms, an Ethics of Care can help
mitigate the most significant and widespread flaws in AI
systems that will impact people's lives in consequential
ways. An AI Ethics of Care has the potential to apply to
areas far beyond predictive policing, and can inform
many applications of AI for consequential decisions.

Acknowledgment
This work was supported in part by a Beneficial AI
research grant from the Future of Life Institute.

Author Information
Peter Asaro is Associate Professor and Director of Graduate Studies in the School of Media Studies at The New
School, New York, NY. He is also Visiting Professor at the
Munich Center for Technology in Society at TU Munich,
and Affiliate Scholar at Stanford Law School's Center for
Internet and Society. Email: asarop@newschool.edu.

References
[1] P. Miller and T.O'Leary, "Accounting, 'economic citizenship' and
the spatial reordering of manufacture," Accounting, Organizations
and Society, vol. 19, no. 1, pp. 15-43, 1994.
[2] S. Zuboff, In the Age of the Smart Machine: The Future of
Work and Power. Basic, 1988.
[3] L. Winner, Langdon, Autonomous Technology. Cambridge, MA:
M.I.T. Press, 1977.
[4] P. Asaro and W. Wallach, "An introduction to machine ethics and robot ethics," in Machine Ethics and Robot Ethics (The
Library of Essays on the Ethics of Emerging Technologies), W. Wallach and P. Asaro, Eds. Routledge, 2017; http://peterasaro.org/
writing/WALLACH%20ASARO%20(Machine%20Ethics%20Robot
%20Ethics)%20.pdf.
[5] P. Asaro, "What should we want from a robot ethic?," Int. Rev.
Information Ethics, vol. 6, no. 12, pp. 9-16, 2006.
[6] D.K. Citron, "Technological due process," University of Maryland
Legal Studies, Res. Pap. no. 2007-26; Washington Univ. Law Rev., vol.
85, pp. 1249-1313, 2007; https://ssrn.com/abstract=1012360.
[7] F. Pasquale, The Black Box Society. Cambridge, MA: Harvard
Univ. Press, 2015.
[8] A. Selbst and S. Barocas, "Regulating inscrutable systems," presented at WeRobot 2017; 2017; http://www.werobot2017.com/wpcontent/uploads/2017/03/Selbst-and-Barocas-Regulating-InscrutableSystems-1.pdf.
[9] R. Caplan, J. Donovan, L. Hanson, and J. Matthews, "Algorithmic accountability: A primer," Data & Society Tech. Rep., Apr.
18, 2018; https://datasociety.net/output/algorithmic-accountabilitya-primer/.
[10] V. Eubanks, Automating Inequality: How High-tech Tools Profile, Police and Punish the Poor. St. Martin's, 2017.
[11] S.U. Noble, Algorithms of Oppression: How Search Engines
Reinforce Racism. New York, NY: New York Univ. Press, 2018.
[12] C. O'Neil, Weapons of Math Destruction. Crown Random
House, 2017.

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

∕

JUNE 2019

http://peterasaro.org/writing/WALLACH%20ASARO%20(Machine%20Ethics%20Robot%20Ethics)%20.pdf http://peterasaro.org/writing/WALLACH%20ASARO%20(Machine%20Ethics%20Robot%20Ethics)%20.pdf http://peterasaro.org/writing/WALLACH%20ASARO%20(Machine%20Ethics%20Robot%20Ethics)%20.pdf https://www.ssrn.com/abstract=1012360 http://www.werobot2017.com/wp-content/uploads/2017/03/Selbst-and-Barocas-Regulating-Inscrutable-Systems-1.pdf http://www.werobot2017.com/wp-content/uploads/2017/03/Selbst-and-Barocas-Regulating-Inscrutable-Systems-1.pdf http://www.werobot2017.com/wp-content/uploads/2017/03/Selbst-and-Barocas-Regulating-Inscrutable-Systems-1.pdf https://datasociety.net/output/algorithmic-accountabilitya-primer/ https://datasociety.net/output/algorithmic-accountabilitya-primer/

IEEE Technology and Society Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - June 2019