IEEE Robotics & Automation Magazine - September 2019 - 12

A ProPublica study found that "black
defendants were far more likely than
white defendants to be incorrectly
judged to be at a higher risk of recidivism, while white defendants were
more likely than black defendants to
be incorrectly flagged as low risk" [3].
These tools go beyond mere image
classification and influence decisions
about sentencing and parole, with
important impacts on people's lives and
freedoms. In addition to reinforcing
discriminatory practices, the company
that created the algorithm was criticized for lacking transparency because
the algorithm is unknown and does not
allow for external auditing to understand and potentially correct biased
outcomes. Search engines have also
been accused of reinforcing bias and
stereotypes. For example, a study of the
Google Ads system found that "setting
the gender to female resulted in getting
fewer instances of an ad related to high
paying jobs than setting it to male" [4].
Beyond data sets, the very nature of
classification algorithms, especially
those based on visual data, has raised
ethical concerns [5]. The mere decision
to define classes can already be discriminatory. For example, when using visual
data to find someone's gender, algorithms typically employ binary classification based on biological sexual traits
(male or female) expressed as visual features. This selection of binary classes
implicitly denies more complex gender
identities with the risk of invisibilizing or
discriminating against gender-fluid or
transgender people. Furthermore, it has
been argued that reducing personality
traits, gender, or emotions to visual features is dangerous, as it potentially
"makes a value judgement about a person based on their mannerisms and the
appearance of their body" [5]. Recent
applications of machine learning have
raised important concerns about technology enforcing prejudicial practices
when aiming to recognize criminals,
sexual preferences, or complex emotions
based solely on visual data. These examples shed light on the potentially complex socioeconomic and cultural side
effects of seemingly value-free applications of recent AI-based technologies.
12

IEEE ROBOTICS & AUTOMATION MAGAZINE

What Are the (Unintended)
Consequences for Robotics?
Many of these technologies are, in fact,
core building blocks of robotic systems
intended to be used in interactions with
humans. It therefore is clear that unsuspected and unintended bias issues will
also be pervasive in robotics applications. A perception system for an autonomous car that detects people in a
significantly different way based on
their skin color or gender would create
an unfair distribution of risks across the
population, with certain groups of people more likely to enter into an accident
than others. Light-skinned versus darkskinned issues have already come to the
fore when autonomous vehicles try to
classify what constitutes a human on
the road [11]. However, the particular
complexity of robotic systems might
also accentuate these issues because
multiple biased algorithms could reinforce one another, severely increasing
the difficulty of detecting bias during
complete system operation. Howard
and Borenstein [6] have discussed in
depth how bias in robotics could lead
to the unfair treatment of certain
demographic groups. They question
how life-critical decisions will be made
not only by self-driving cars but also by
medical or service robots if algorithms
introduce or reinforce certain discriminatory practices. Furthermore, they
argue that algorithms could deepen
unfair surveillance and profiling of
minorities by a robotization of law
enforcement systems, e.g., using computer vision to assist policing practices
could accentuate unfair targeting.
Applications involving human-
robot interactions are especially vulnerable, as the feedback nature of
such interactions could amplify biased
responses of the robotic device in ways
that are difficult to detect. In particular,
if the interaction creates an issue, it
might be difficult to distinguish detrimental behaviors before field deployment. While trust is an important and
desirable factor for human-machine
interactions, strong user confidence and
reliance on robotic systems could
potentially worsen the detection of
unfair behaviors. One could legitimately

SEPTEMBER 2019

ask whether biased machines will not
only perpetuate the discriminatory
practices present in all societies but
also introduce unforeseen problems
that will make them even worse-and
at an unprecedented scale-as robotic technologies increasingly interact
with humans.
The Need for Multidisciplinary
Approaches to the Problem
The reasons for biased outcomes often
are very complicated. A lack of diversity in the training data sets used in
machine learning is a central cause.
For example, the Labeled Faces in
the Wild data set contains more than
15,000 images of faces but as few as 7%
of the images are of black people [7]. It
has been argued that this lack of data
diversity stems, to some extent, from
the actual lack of diversity in many AIrelated fields, creating a feedback loop
of bias [5]. The available data also can
be intrinsically skewed and not usable,
as in the hiring tool example. It is not
clear how certain data sets can be
diversified, because historic data may
contain the stereotypical patterns that
one would expect to correct. Furthermore, explicitly removing the biasing
attributes from the learning algorithm
might not be sufficient due to the correlations with other features that are
often hidden, highlighting the additional difficulty of choosing the appropriate input features. Moreover, we all
have unconscious or implicit biases
that might further influence algorithm
design and outcomes.
The consensus in the machinelearning community is that bias is an
issue that needs to be specifically addressed. From a technical standpoint,
research on algorithmic fairness proposes formal methods to mitigate the
risk of bias. For example, anticlassification strategies ensure that classification results do not depend on protected
attributes, such as gender or race, and
classification parity ensures that evaluation performance is equal across
groups defined by protected attributes.
However, there are fundamental technical limitations to a mere algorithmic approach [8]. Indeed, it has been

IEEE Robotics & Automation Magazine - September 2019

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - September 2019