IEEE Computational Intelligence Magazine - February 2022 - 74

requirements on the basis of the medical imaging use case. The
conclusion was that current trends in machine learning introduce
a level of complexity that is hard to reconcile with existing
legal requirements, especially with regard to human
legibility of explanations for non-expert data subjects. Furthermore,
the paper reasoned that the quality of possible
explanations might not be judged as adequate under the provisions
of the GDPR. In this work, these preliminary findings
are strengthened by systematically exploring the technical
challenges linked to the usage of AI techniques in decisionmaking
systems through two use cases. The novelty of this
study lies in the interdisciplinary exploration of technical and
legal issues to tackle a number of open questions and more
general problems, including:
❏ What are the main challenges to providing explanations
regarding the realities of contemporary AI?
❏ What are legal and technical implications if explanations
cannot be provided, or can only be provided partially
according to the legal requirements from the GDPR?
❏ How could a right of explanation be embedded and
enforced in the context of auditing and regulatory processes
such as an impact assessment?
❏ How could a regulatory setting be devised to compensate
for the technical limitations?
The outline is as follows: Section II introduces the necessary
background on legal requirements of automated decision-making
and AI explainability techniques. Section III presents the
core analysis of the challenges using the use cases as illustrative
examples. Section IV exposes the doubts about the feasibility of
a monolithic approach to requiring explanations in the future
AI-based environment, and proposes some alternative
approaches to mitigate the situation.
II. Background
In this section, background elements regarding the legal
requirements set out in the GDPR on automated decisionmaking
are provided, as well as an overview of the field of
Explainable AI (XAI) that aims to complement AI systems with
meaningful explanations of their mechanisms.
A. Legal Requirements on Automated Decision-making
The GDPR [3] came into application in May 2018 in the
European Union (EU) and introduced meaningful novelties in
EU legislation about automated decision-making systems based
on the processing of personal data (information related to an
identified or identifiable individual). Considering the large use
of personal data in AI applications and the specific data protection
rules about explainability and accountability of automated
decisions, it would appear that the GDPR is, currently, the most
advanced, robust, and comprehensive legal tool to regulate the
use of personal data in AI. One of its main goals is to protect
fundamental rights and human rights of individuals when they
are subject to-among others-profiling and automated decision-making
[10]. Looking at the text of the GDPR, the definition
of profiling given in Article 4(1)(4) and the general
74 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2022
principles of transparency, lawfulness, fairness, accuracy, and
accountability (Article 5) seem very relevant elements for trustworthy
AI applications.
Moreover, Article 22 provides a right not be subject to decisions
based solely on automated decisions that produce legal or
similarly significant effects, with three exceptions [11]: the consent
of the data subject, the necessity for the performance of a
contract, or an EU or national law that regulates specific cases
of automated decision-making. In case these automated decisions
are taken, the data subject should have always the rights
to: contest the automated decision [12]; express their own view;
have a human intervention in the automated decision-making
systems [13]. In addition to these rights, recital 71 clarifies
that-depending on the level of risk of the algorithmic application-the
safeguards for data subjects should also include the
auditing of the algorithm and the right to receive an explanation
about the individual decision taken. The existence and
modalities of such a right to an explanation have triggered an
intense discussion among legal and technical scholars [4], [14]-
[16] and led to its implementation by several EU member states
in their national laws [17]. In addition to these explanation
requirements, Articles 13(2)(f), 14(2)(g), and 15(1)(h) of the
GDPR require that data subjects are informed about the existence
of automated decision-making processes and about the
underlying mechanisms (logics) behind the automated decision-making
performed, as well as the significance and potential
consequences of such processing [18].
Different levels of explanation can be identified: a general
one about the logic of the AI system as a whole, and more granular
explanations about how the AI application works in particular
situations (e.g., for some specific cluster or group of
individuals), or even in individual cases. These kinds of explanations
can be also relevant to better exercise the right to contest
the output of the algorithm and the right to human intervention.
Indeed, the guidelines released on this topic by the European
Data Protection Board (EDPB) state that human
intervention implies that the human-in-the-loop should refer to
someone with the appropriate authority and capability to
change the decision [2]. It is clear how a requirement of technical
explainability is relevant for these envisaged safeguards.
Human supervision can only be effective if the person reviewing
the process is in a position to assess the algorithmic processing
carried out. This implies that such processing should be
understandable. Furthermore, explainability is also key to ensuring
that data subjects are able to express their point of view and
are capable of contesting the decision. As stated in the EDPB
guidelines, data subjects will only be able to do that if they fully
understand how the automated decision-making was made and
on what basis. In summary, considering the different levels of
risk that automated decision-making can imply for rights and
freedoms of data subjects, some scholars proposed the following
multi-layered explanation model [16]: for low-risk scenarios, a
general layer of explanation about the algorithmic functioning
might help (inferred from Articles 13-15 GDPR); for mediumrisk
scenarios, an intermediate and group-based explanation of
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
