IEEE Technology and Society Magazine - June 2015 - 54

Consider the following thought experiment: Steve is
travelling along a single-lane mountain road in a selfdriving car that is fast approaching a narrow tunnel.
Just before entering the tunnel a child errantly runs into
the road and trips in the center of the lane, effectively
blocking the entrance to the tunnel. The car is unable
to brake in time to avoid a crash. It has but two options:
hit and kill the child, or swerve into the wall on either
side of the tunnel, thus killing Steve [23]. If the decision
must be made in milliseconds, the computer will have
to make the call [24], [25]. What should the car do?
Similar to the situation involving Jane's end-of-life decisions and the state of Jane's ICD (should it remain active
or be deactivated?), this ethical dilemma has no objective
answer. But the proxy analysis framework applied previously to the ICD helps us to shed some light on questions
surrounding this driverless car scenario. Regardless of
which path the SDC takes, we can consider the SDC a
moral proxy acting on behalf of whoever set it to take
that path. If, for example, the engineers at Google programmed the car to keep going straight, in other words to
always protect Steve's life in situations where the vehicle
is not legally at fault for the situation, then we can say that
the SDC is acting as moral proxy on behalf of Google. But
this is morally problematic. It might be the case that Steve
considers himself morally obliged to risk his own life in
such situations, especially ones involving children. Thus,
Steve would find himself subject to a paternalistic relationship: a moral proxy acting on behalf of Google would
confound his moral preferences. Indeed, according to the
proxy model it is always paternalistic to keep Steve out
of the decision-making loop, regardless of whether the
car's programming corresponds to his moral preferences.
Steve is the morally appropriate decision-maker in this
driving context since his life, not the life of the manufacturer, is directly at risk. It follows, as in the healthcare context, that his is the autonomous decision the proxy ought
to represent, not Google's.
From a design perspective, this suggests that SDC
owners (or passengers) ought to be morally responsible for setting the SDC to act one way or the other
in situations like the one described. Designers are not
the appropriate decision-makers in these scenarios,
and so should reasonably strive to build options into
SDCs allowing the choice to be left to the user. As is
the case in medical contexts, SDC users ought to be
informed of the potential moral consequences of their
technology choices. Users should then be asked to make

autonomous decisions regarding certain cases where it
is reasonably foreseeable that the technology will provide material answers to moral questions, in this case
risking either the owner's life or the lives of others in
particular use contexts. Making that decision on behalf
of the user risks subjecting him to a paternalistic relationship in which his autonomy is unreasonably fettered.

Devices Providing Material Answers
to Moral Questions
STS provides numerous examples of how technological
devices can provide material answers to moral questions in use contexts. When a device reifies a moral
decision it instantiates a moral proxy relationship - a
role much discussed in bioethics - acting on behalf of
whoever set it to operate in that particular mode. There
are established ethical norms surrounding proxy decision-making in healthcare and bioethics. We ought to
apply those norms to technological moral proxies. Users
should have their autonomy reasonably maximized; the
devices ought to act explicitly on their behalf. Designers
can maximize user autonomy by designing devices such
that users are better informed about the proxy implications of the device, and such that users make explicit
choices regarding the settings that reify moral decisions.
Placing requirements on designers may seem
misplaced. Why should we? The short answer is that
designers - engineers for the most part - are already
designing norms into their devices as the ICD example
suggests. Whatever decision an engineer makes about the
ICD's switch carries moral implications. Design is a social
and ethically loaded activity, through and through [2]-[4].
What engineers should want to avoid is accidental
paternalism in design. Engineers are inescapably linked
to the settings and features they embed in their devices,
so they should understand the moral dimensions of
those links, and design the devices explicitly using
some guiding framework. Accidentally creating devices
that subject users to paternalistic relationships is undesirable for users and for engineers.
Moral proxies and informed consent practices provide
a conceptual framework that can be applied to the design
process to help avoid accidental paternalism. Though
such a framework does not promise to help predict or prevent all morally problematic design decisions with proxylike implications (what process could accomplish such a
lofty goal?), recognizing the applicability of the proxy analysis framework to design is a step toward better design.
There is, of course, a practical gap in my
argument. One should ask, how should we
actually apply these concepts in the design
process? My goal in this article has been to
lay the theoretical underpinnings for a practical approach to performing a proxy analysis.

IEEE Technology and Society Magazine

∕

june 2015

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - June 2015