IEEE Technology and Society Magazine - June 2017 - 73

Rule 5 - "People who design, develop, deploy, promote, or evaluate a computing artifact should not
explicitly or implicitly deceive users about the artifact or its foreseeable effects, or about the sociotechnical systems in which the artifact is embedded."
(emphasis added)
How transparent will companies be about how the
self-driving car is designed to behave, especially in
dynamic or dangerous traffic situations? Market forces
and other forms of competition may pressure engineers
and companies to present the car as "risk free" or "of
minimal risk" to human drivers and passengers. This is
already happening to some degree [56]. Yet Google has
recently admitted that one of its cars could be blamed
for an accident [57].
The first reported fatal accident involving a self-driving car occurred in May 2016. While under the control of
its autopilot system, a Tesla car crashed into a tractortrailer that was making a left turn in front of the car. A
Tesla blog post suggested that since the tractor-trailer
was white, it might not have been visible against the
brightly lit sky to the car's autopilot system. That same
post also stated, "Nonetheless, when used in conjunction with driver oversight, the data is unequivocal that
Autopilot reduces driver workload and results in a statistically significant improvement in safety" [58].
Lucas Merian notes however that "The problem for
Tesla has been that while its Autopilot ... offered some
[SAE standards] level 3 automation, there was no way to
force a driver to retake control of the vehicle; that has
resulted in several documented accidents" including a
fatal one [59]. Merian goes on to argue that "[T]he problem ... hasn't necessarily been that Tesla's Autopilot ...
isn't performing as promised, but that drivers place too
much confidence in it and take their hands off the steering wheel and their attention from the road." Following the
accident, Elon Musk asserted that Tesla is implementing
changes to its autopilot system that will purportedly prevent this type of accident from recurring, including limits
on how long a driver's hands can be off the wheel and
improved radar for recognizing obstacles [60].
One could argue that Tesla's response to the accident is consistent with a number of "the Rules." The
blogger's and Musk's statements seem to be offering a
defense that Rule 5 is being upheld by the designers
of the self-driving car as long as the public has a reasonably accurate view of the associated risks. The
blogger also speaks to Rule 1 in so far as Tesla is
claiming that it has considered the risks and is confident that their self-driving car reduces (although clearly does not eliminate) the chance of harm to the
public. Rules 1 and 3 are both arguably addressed
by Musk's remarks concerning recognition by Tesla
that there is a driver in the loop who needs to be
■

june 2017

∕

considered by the designers. Nevertheless, the fact
that such accidents have occurred with automation in
the range of Level 2-3 suggests, as we argue below,
that vehicles in the range of Level 3-5 automation
should not be permitted on the road until more thorough testing has been conducted.

Our Proposal
In the interest of public well-being and safety, designers,
testers, managers, and others should sincerely engage with
"The Rules" and contemplate their implications for decision
making regarding self-driving cars. At an individual level,
each designer should consider his/her ethical obligations in
terms of creating safer technology. One approach that
incorporates this type of thinking is value-sensitive design,
which encourages designers to consider how the user's
cherished values, such as autonomy, can be upheld while
in the process of creating their technologies [61]-[62].

People who design, develop, deploy,
promote, or evaluate a computing
artifact should not explicitly or
implicitly deceive users about the
artifact or its foreseeable effects.
It is also essential that relevant professional communities become collectively involved in a deliberative and
reflective process. More specifically, engineering, computing, and other communities should engage in a variety of activities related to anticipatory ethics, including
how they can take steps to minimize the impact of system failures in self-driving cars. This could be akin to an
"Asilomar-like" activity. Asilomar was a conference in
1975 where scientists gathered to discuss recombinant
DNA research and then developed voluntary guidelines
to help protect the public.
Unlike Asilomar, however, efforts concerning autonomous vehicles should be structured so as to include the
views of stakeholders from outside of the science and
engineering community [63]. This activity could be patterned after efforts being witnessed in other realms of
emerging technology. For example, the BEINGS conference gathered together scientists, philosophers, lawyers, industry representatives, and others to discuss the
ethics of gene editing technologies [64]. Stakeholders
from across the globe have organized numerous events,
including United Nations meetings, to address concerns
about the use of military robots [65]-[66].

IEEE TEchnology and SocIETy MagazInE

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - June 2017