IEEE Computational Intelligence Magazine - February 2022 - 83

Accordingly, in such situations, a DPIA should be necessary:
the data controller should describe the data processing, assess
the necessity and proportionality of it, analyze risks for data
subjects (e.g., risks of discrimination, violation of the right to
health, to freedom of thought, of movement, attack to their
digital identity, etc.), and seek for the data subjects' opinion on
the data processing. This procedure will also imply the description
of the logic of the AI-based system, as well as the assessment
of its impact on the fundamental rights and freedoms
relevant for the application and the ways of mitigating it [70].
Making this report public (at least in some parts), would also
significantly help to reach a better level of transparency and justification
of AI applications [10].
The risks to rights and freedoms1 might concern, in the
credit scoring scenario, a) the discrimination of some social
groups (based on ethnic origins, gender, sexual origins, social
conditions) in the access to loans that might be embedded in AI
biases; b) the automatic exploitation of individuals in situations
of financial vulnerabilities or in situations of gambling addictions
through systems of unsustainable interest rates; c) the economic,
social and psychological disadvantages of families of
individuals in accessing loans at unfavorable conditions (resulting
in adverse effects to the fundamental right to health or even
to conduct business). Mitigating these risks might be done also
without requiring advanced explainability techniques: for example,
the data controller could check the presence of eventual
biases against social minorities or against gambling addicted
people in the training data sets of the credit scoring systems. In
addition, the data controller could check on a regular basis the
effects of credit scoring outcomes to see whether it led to unfair
economic, social or psychological damages to the consumers,
seek for the opinion of the representatives of the data subjects
(as Article 35(9) GDPR requires), or prepare periodical reports.
Regarding the medical imaging scenario, the risks to rights
and freedoms might include, a) the discrimination of some social
groups (based on ethnic origins, gender, sexual origins, social
conditions) in their access to COVID-19 medical treatments2;
b) false negatives that could lead to serious health damages to the
patient, and to the group of people that might be infected by
him/her; c) false positives that could lead to unnecessary medical
treatments (that might both damage the health conditions of the
patient and diminish the availability of drugs to people that
might need them in a pandemic), and to psychological suffering
of the patient and of his/her family; etc. Even in this case,
addressing and mitigating these risks might be done without
opening the 'black-box' of the AI application. For example, the
data controller could check the presence of eventual biases
against social minorities in the training data sets of the COVID19
detection tool. The data controller can also check the effects
of COVID-19 detection outcomes on a regular basis to see
1About the concept of " risk to a right " in the GDPR provisions about the DPIA see
[71]-[75].
2Although this might appear as a mere hypothetical case, many studies in the US
proved discrimination of health-related algorithms against black people. See for example
[76].
whether it led to false positives or false negatives and whether
these false diagnoses refer to people belonging to some particular
minorities (ethnicity, gender, etc.). In both cases, when these
problematic aspects are detected, the data controller could provide
some organizational mitigation measures, for example a second-step
human-mediated decision in 'crucial' cases where the
user belongs to a situation that has been observed (during the
algorithmic auditing) to be at high risk of unfairness, inaccuracies
and adverse effects in general3. In addition, the data controller
could organize some periodical questionnaires on people
who used this tool to check whether the system produced further
damages, or perform some simulations to see whether particular
people are discriminated or anyway adversely affected.
C. Proposal: Justification of AI-based Decision-making
Systems in the Context of Transparency Requirements
AI systems can be judged from different perspectives, considering
various aspects such as performance, robustness, transparency,
or explainability. So far, the discussion regarding algorithmic
explanations has focused on the latter, exploring the right to
explanation. However, a broader consideration of those aspects
becomes valuable in cases where no sufficient technical explanation
can be provided according to Article 22(3) of the
GDPR. Therefore, we argue that within the scope of a full AI
risk assessment, as outlined in Section IV-B, one should in general
not focus only on one particular risk. As an illustration,
high performance in a critical application-such as in the
medical imaging use case, where performance could save life-
might warrant the risk of losing human legibility of results.
Beyond the trade-off between performance and explainability
discussed in Section III-B, the robustness of AI systems is
also highly intertwined with explainability. It is reasonable to
assume that a robust model, systematically and properly
checked, will yield more robust and trustworthy explanations,
even if it might not be directly human-legible. This provides a
crucial link between explainability and the broader topics of AI
safety and cybersecurity. In the context of the latter, another
trade-off might even be introduced between transparency and
security, by increasing the attack surface against models [78] in
providing full technical transparency, including ways to compromise
the validity of explanatory methods [79]. It could also
be argued that for an AI model lacking in explainability, conducting
such proper technical system performance validations
and increasing the overall robustness could be enough to provide
information about the logics of the system, at least for a
technical expert or an auditing process.
Likewise, addressing fairness issues in automated decisionmaking
systems is obviously connected to addressing
infringements on individual rights, as are the issues surrounding
the legal requirements on explanations. Both
fields could be jointly considered, and many explanatory
methods are seen as instrumental to detect biases and unfair
3This approach might be criticized because it would imply to use a bigger amount of
sensitive data in order to avoid discrimination. However, see the reply in [77].
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 83
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
