IEEE Technology and Society Magazine - December 2019 - 25

of the Chinese Embassy incident,
mistakes are made at this strategic
level, but the line of accountability
is quite clear.
The second layer of target identi-
fication resides in the design of an
autonomous weapon. Once a legiti-
mate target is certified by a human
decision maker, the question must
be asked as to whether an autono-
mous weapon can correctly identify
and engage a target with better than
human odds. Such a weapon would
need to adapt to changing battle-
field conditions to respect rules of
engagement. In the case of the Tom-
ahawk missile, if a target is static,
well mapped, and within the rules of
engagement, it meets this criterion.
Unfortunately, only a small percent-
age of targets fall into this category. If
dynamic and emergent targets need
to be prosecuted, any autonomous
weapon system used in this manner
should have to prove it is significantly better than humans in similar
circumstances. Better in this case
means better at correctly identifying
the target and better at responding to
emergent events and respecting the
rules of engagement. Unfortunately,
as illustrated in Figure 1, computer
vision systems that rely on machine
learning techniques simply do not
meet this standard at this point in
time or in the near future.
This issue of autonomous system
certification, i.e., that such a system
can be shown to perform better than
a human at a complex safety-critical
task, has not been solved for either
military or civilian systems, and is
a topic of intense debate. Because
such systems rely on probabilistic
reasoning, often using black box
algorithms like convolutional neural
networks, we currently cannot pre-
dict with high levels of assurance
how such systems will perform, es-
pecially in unforeseen conditions.
Engineers do not know how to test
these systems for possible errors,
DECEMBER 2019

either errors of commission or omis-
sion. For example, engineers do not
know how to predict when a com-
puter vision algorithm will make a
misidentification such as in Figure 1.
Significantly more work is needed
across multiple industries that wish
to leverage autonomy, including
not just autonomous weapons but
driverless cars and surgical robots,
to better understand how we can
determine prior to a technology's
deployment whether it truly is likely
to exhibit better than human levels
of performance [4], [5].
Instead of advocating for a ban
on such weapons, I believe the inter-
national community should demand
that if autonomous offensive weap-
ons are used in the future with
embedded artificial intelligence, they
must meet very strict certification cri-
teria, both at the strategic layer in tar-
get selection and in the design layer
with autonomous target identifica-
tion. Such weapons systems should
be proven through objective and
rigorous testing, and should dem-
onstrate an ability to perform better
than humans would in similar cir-
cumstances, with safeguards against
cybersecurity attacks. Currently in
the United States, manufacturers
of military weapons are indemni-
fied against accidents on the battle-
field. It is this policy that should be
banned, especially for autonomous
weapon systems.
Given the potential misuses and
abuses of autonomous weapons
technologies, the burden of proof of
performance and safety should fall
on the shoulders of industry as well
as the military branches who buy
their weapons. Both these groups
should be required to certify to as-
yet-determined international stan-
dards that such weapons can be
used safely and responsibly, and
with proof of strong safeguards
against hacking attempts. Increas-
ing accountability in this very spe-
∕

IEEE TECHNOLOGY AND SOCIETY MAGAZINE

cific way can work towards providing
checks and balances needed to
guard against irresponsible autono-
mous offensive weapon use.
Often those against the use of
autonomous weapons will compare
this technology to that of nuclear
weapons, asserting that the very
existence of humanity is threatened
by killer robots [7]. Proponents of
nuclear weapon bans cite the lack
of proportionality and distinction,
central tenants of Just War Theo-
ry, as reasons for a ban, which are
similar arguments for banning land
mines. Autonomous weapons are
almost the opposite in that, as dem-
onstrated by the Tomahawk missile,
they can be far more precise and
proportional than most any other
weapon in the existing U.S. inven-
tory. It may be possible, once the
computer vision and cybersecurity
issues are addressed, for an autono-
mous weapon to close on a target,
detect the presence of a child in the
last few seconds of an attack, and
then divert itself to a less harmful
detonation point. This is not pos-
sible for the same missile under
current versions of human control.
Moreover, while non-state actors
could abuse such a technology, just
as for nuclear weapons and land
mines, formalized bans do not stop
rogue actors.
Indeed, an argument could be
made that if we know humans are
prone to errors in many weapons
launch cases and are unlikely to
provide actual meaningful human
control, then it would be more ethi-
cal for commanders to launch a
certified autonomous weapon. We
have achieved this capability today
for static targets (like buildings)
in that it is better for a Tomahawk
missile to be launched to take out
a legitimate military target than it
is for a human pilot. One possible
derivative effect of such technolo-
gies could be reduced inclination

25
IEEE Technology and Society Magazine - December 2019

Table of Contents for the Digital Edition of IEEE Technology and Society Magazine - December 2019
