IEEE Computational Intelligence Magazine - February 2022 - 82

Even the elementary machine learning systems implemented in
the credit scoring use case may exhibit these contextual limitations,
while in the context of a computer vision problem such
as the medical imaging scenario developed in the COVID-19
use case, providing generally legible explanations may be simply
impossible. In addition, the possibility to give a satisfactory, fair,
and transparent explanation is often considered to depend on
the possibility of showing a causal link between the input data
and the final decision [66], which clearly has been illustrated to
be fundamentally limited for current AI systems.
Explanations-in the traditional sense-are not only sometimes
problematic, but also not sufficient to make AI socially and
legally 'desirable'. In particular, several scholars reflected upon the
" transparency fallacy " of algorithmic explanation [67], i.e., the
risk that even a meaningful explanation could be not effectively
received or understood by the data subjects (due to its technical
nature or to the limited attention, interests or-even temporary-cognitive
capabilities of the data subject) [68]. Thus, in line
with our preliminary analysis [8], the quality of possible explanations
in some cases might not be judged adequate under Article
22(3) of the GDPR. Ultimately, adopting a strict approach, this
would mean that either more technologically advanced (and
possibly obscure) forms of decision-making should be prohibited
because their decisions are unexplainable, or, conversely, that we
should tolerate AI-based decision-making systems that do not
formally respect the transparency duties of the GDPR. These last
considerations may lead to an impossible choice: technologically
advanced decision-making systems are prohibited because they
cannot comply with the GDPR explainability requirements; or
AI-based decision-making systems that do not formally respect
the transparency duties in the GDPR are tolerated.
Instead, a possible third way might be a technical and practical
non-monolithic interpretation of a right to explanation in
the context of automated decision-making systems. In order to
do that, the right to explanation in the broader picture of the
GDPR should be better contextualized. In particular, Article
22(3) and recital 71, when addressing the possible safeguards to
reach accountable automated decisions, do not focus mostly on
the right to explanation, but on a varied set of tools (including
the right to contest, human in the loop, and algorithmic auditing).
Further, in Articles 13-15 the " right to obtain meaningful
information about the logic involved " in a system does not
explicitly require this information to be a full explanation of the
decision-making system as exemplified in our case studies.
Beyond Articles 13-15 and 22 (and recital 71), the GDPR contains
several notions that might influence the interpretation of
accountability duties also in case of automated profiling and
explanations. Data protection principles in Article 5 include the
fairness principle, the lawfulness principle, the accuracy principle,
the purpose-limitation principle, the data minimization and
the storage limitation principle, the integrity and confidentiality
principle, and the accountability principle, which seem all very
relevant requirements towards more desirable AI.
In our view, in cases where the best possible technical
explanation is not satisfactory, further tools should be
82 IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE | FEBRUARY 2022
implemented too. Firstly, they might rely on the disclosure of
meaningful information about a Data Protection Impact
Assessment (DPIA) on the algorithmic decision-making system
(see Section IV-B). Secondly, apart from assessing the potential
impact, a different set of tools, technical developments and
approaches could be considered in order to mitigate the identified
risks, and embed a potential lack of explanation in a framework
of additional measures. These could be the result of a
justification test on the algorithm, based on the mentioned data
protection principles given in Article 5 (see Section IV-C).
B. Proposal: Explanations and the Data
Protection Impact Assessment (DPIA)
As affirmed in the previous section, the impact assessment of AI
applications might be not only a desirable solution, but also a feasible
way to overcome many problems about the inscrutability of
black-box systems, while reaching an adequate justification of
algorithmic decision. The impact assessment model proposed by
the GDPR is the Data Protection Impact Assessment (DPIA)
that could also be used for AI applications. The DPIA, as regulated
in Article 35 of the GDPR, is a tool to describe, assess, and
mitigate the risks for data subjects resulting from the processing
of personal data. As noted in [69], the DPIA comes in addition to
the obligation for data controllers to manage risks, consisting in
their identification, their analysis, their estimation, their evaluation,
their treatment, and their regular reviews. In the case of possible
infringement of the rights and freedoms of data subjects, a
DPIA must be carried out. The EDPB indicated ten indices for
high-risk data processing: evaluation or scoring, automated decision-making
with legal or similar significant effect, systematic
monitoring, sensitive data or data of highly personal nature,
large-scale data processing, the combination of multiple data sets,
the presence of vulnerable data subjects, the use of innovative
technologies or organizational measures, extra-EU data transfers,
and the limitation of data subjects' rights. In summary, applying
the DPIA framework to AI systems would be essential to protect
fundamental rights and human rights of individuals [70]. Some
scholars have even tried to elaborate a DPIA model that could
take into account all concerned human rights of individuals [10].
Both use cases presented in this paper clearly mark such highrisk
situations for which the data controller would need to carry
out a DPIA. For the credit scoring use case, the banking institution
is processing financial records of customers alongside personal
information (location, employment, possibly health, etc.)
that might be considered either as sensitive data or as data of a
highly personal nature, used for evaluation or scoring. Automated
decision-making based on these data may have legal or similar
significant effects, and this may prevent the data subject from
exercising a right or using a service under a contract. Even more
pronounced is the relevance of the medical image case study: the
hospital is processing patient's health data, which are considered
sensitive data. In addition, patients may be considered as vulnerable
data subjects, and the data processing may be done on a large
scale. The use of multiple data sets and of innovative technologies
could also apply to many machine learning based systems.
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
