IEEE Technology and Society Magazine - June 2019 - 51

this paper will contribute to the discussion of the ethical
design of AI, especially as AI comes to be applied in an
increasing number of socially significant and ethically
consequential decisions.
While not every AI system developer can become an
expert in the application domain of their techniques, the
basics of critical data analysis should be taught alongside statistical techniques and machine learning techniques. In particular, system designers should be adept
at recognizing the necessary characteristics of an adequate dataset, and what can and cannot be reasonably
drawn from a given dataset. In many cases, only domain
experts will have the kind of cultural knowledge to identify exogenous influences. This fact supports a systems
design approach that includes domain experts as well as
critical social scientists as members of design teams,
and recognizes and respects the necessity of their
expertise in shaping the ultimate system design [56].

nor be available means for a law enforcement officer. To
some extent there is discretion on the part of law
enforcement, prosecutors, and judges as to how to
appraise and categorize such a crime - and they may
take factors into account other than the strict value of
the property. But once categorized, that discretionary
nature tends to be erased - the crime becomes
defined through its given category, documented and
entered into data collection systems. AI systems designers need to be sensitive these types of processes.
Indeed, understanding data collection, and critical data
representation issues should be integral to computer
and information science education. Taking care in the
design of AI means being able to determine what an
adequate dataset is, and being able to think critically
about how to define it, and what the implications of various choices of categorization are. How best to do this,
in general, is a matter for further research.

Models Matter

Putting AI Into Practice

A dataset on its own is just a collection of numbers
delimited by some kind of file structure. Even decisions
as to how to represent a data field with a number -
binary, integer, real, pointer, formula - can have consequences for how that data gets processed. Numbers are
abstract values, which are then represented by digital
numerals within computational systems. How they are
numerically represented can matter. But often it is far
more important how we choose to represent the world
through numbers. Even when we are simply "counting"
things in the world, we are also engaged in processes of
classification and categorization. The data "model" that
a system employs involves myriad representational
choices, and seeks to serve various purposes [57].
The most obvious case in law enforcement is to characterize the law, and represent violations of the law. But
there are many possible computational models of any
given set of legal rules and codes, and they may not
always represent the same mappings of events in the
world as computational encodings.
Consider the case of CompStat crime underreporting
discussed above. We could look to New York Penal Law
§155.05 and §155.25 for a definition of "Petite Larceny"
which is theft or withholding of property valued at less
than $1000 (and not a firearm, automobile, or credit
card) [58]. What if a bike has been stolen, which cost a
little more than $1000 when it was new, but it is used
and would likely not sell for that much, nor would an
insurance company compensate its loss for more than
$1000? Determining the appropriate crime requires estimating the value of the property. This is a non-trivial categorization - an auction might determine the current
market value, or a bike sales expert might be able to
give an appraisal, but these may not agree on the price,

The discussion so far has focused on input - how data
is structured and collected. But the presentation of data
analysis, and its impact on individual and institutional
practices must also be taken in account. A good example of such an issue can be seen in the use of the SSL
by Chicago police. In principle, the SSL could have been
used to recruit youth for the One Summer program. The
choice by precincts and officers to use the list for "custom notification" and for "heat lists" following crimes is
not disconnected from the design of a system like SSL.
While data scientists and software engineers may wish
to wash their hands of responsibility for how officers
actually use their tools, they cannot. At the very least
this constitutes a sort of negligence and failure to warn.
Many officers were not properly or fully informed of how
the list was put together, and held mistaken and problematic understandings of what it was and how it
worked. The officers also lacked training, guidance, and
direction on how to use the system, if indeed there ever
was a comprehensive plan as to how to deploy and use
the system. These factors surely contributed to its misuse, and all but guaranteed its ineffectual use.
An Ethics of Care approach ought to ensure that the
operators of AI systems and users of data they generate
are aware of the scope and limitations of those systems. It may be too much to expect them to fully understand the computational techniques - indeed even AI
experts may find the performance of certain machine
learning systems inscrutable. But this does not mean
that people who use these systems can be ignorant of
what the system can and cannot do, how reliable it is,
and what its limitations in representing the world are.
Designers also need to be aware of the context in
which AI systems will be deployed and used. It should

JUNE 2019

∕

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

IEEE Technology and Society Magazine - June 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - June 2019