IEEE Computational Intelligence Magazine - February 2022 - 77

on the input data, for example as general information for an
expert ex-ante, or, to justify a decision ex-post on an individual
applicant's loan in response to a contestation.
Use Case 2: Medical Imaging
Computer-aided software has been used for decades in the medical
environment to support medical professionals. The recent
progress of AI suggests that these systems may become more and
more autonomous in their decision-making, and help to automate
some aspects of the medical decision-making process that are typically
done by medical staff, to aid the diagnosis and facilitate care
handling of patients. The growing integration of AI capabilities in
automated decision-making gains particular relevance in light of
the current COVID-19 pandemic, during which an uptake of
AI in the health sector has been observed [51]-[56] with the aim
to compensate for the limitations of health systems to handle
exceptional high-stress situations. This includes various forms of
automated triage consisting in defining the priority of a patient for
immediate intensive care. These works are part of a bigger trend in
machine learning to provide detection tools that not only achieve
human performances, but also go beyond and are able to capture
weak patterns in complex images not detectable by humans.
As introduced in [8], the following hypothetical medical
scenario is considered:
A patient with COVID-19 symptoms admits himself/herself
to the emergency ward of a hospital during a severe, ongoing local
outbreak of the COVID-19 pandemic. The nurse tells the patient
that blood tests indicate a possibility of a COVID-19 infection.
Normally, such patients would be admitted after decision of a medical
doctor to the intermediate care ward since a pneumonia induced
by COVID-19 is likely to cause severe and rapid worsening of the
patient. However, no doctor is available for at least several hours to
make the decision. Instead, the patient could decide to let an automated
decision-making system that has been exceptionally authorized
to decide from a chest X-ray image, whether he/she is likely
suffering from COVID-19.
Receiving an automated diagnosis of COVID-19 falls
under the scope of Article 22 as it is a decision based solely on
automated processing and on profiling, which produces legal
effects or similarly significant effects on the data subject. In particular,
at least two effects can be identified: the psychological
effects deriving from a diagnosis, and the medical consequences
of such diagnosis (access to further healthcare, self-quarantine,
etc.). This AI application might be allowed under Article 22
because either it is based on the explicit consent of the data
subject, or it is " authorized by Union or Member State law " .
Accordingly, all measures mentioned in Article 22(3) should be
implemented by the data controller, and the data subject has a
right to receive a " general explanation " before the processing
and upon his/her eventual request at a later stage. As stated in
Section II-A, the higher the risk, the more specific and comprehensive
the explanation of the automated decisions should
be. Thus, since COVID-19 automated detection/diagnosis
might be considered a data processing producing high risks for
the rights and freedoms of individuals, the data controller
might be required to implement stronger safeguards, including
the right to receive an explanation about the decision reached.
The technical implementation of such a decision-making
system is based on the deep learning architecture ResNet-50
[57], which was trained for COVID-19 detection following a
methodology described in [56]. The automated decision-making
system returns a probability of the patient being infected
based on an X-ray image of the chest. The system achieves a
global accuracy of 99% on previously unseen data, with a sensitivity
of 85% to COVID-19, which is in line with published
results for this kind of models [56]. A complementary study presenting
a set of typical multi-layered explanations for this technically
involved scenario is given in [8], providing examples
ranging from informative ex-ante explanations of the system to
counterfactual ex-post explanations of an individual decision.
A. Challenge 1: The Trend to Complexity
in Machine Learning
In machine learning, a clear trend to higher complexity can
be observed, with three major causes:
❏ data sets feeding AI systems are getting more and more heterogeneous
and high-dimensional;
❏ models are made of compound architectures including a
growing number of parameters;
❏ algorithms and techniques used for the development of
models are getting increasingly sophisticated.
The use of XAI techniques to generate sound explanations
adds an additional layer of complexity, as they can be themselves
hard to understand.
1) Complexity of Data
Data sets are often referred to as the new oil to highlight their crucial
role in the development of AI systems. Although relevant in
some contexts, this analogy fails to transcribe the complexity and
diversity of data (images, sounds, texts, tabular data, graphs, etc.) that
significantly differ from the multi-purpose crude oil. Hence, a data
set is only helpful to solve a limited range of applications in a given
context. Historically, tabular data sets, consisting of entries composed
of pre-defined text, value or category fields, have been predominantly
used in early machine learning applications. The credit scoring
data set is a typical example of such type of data (see Figure 2a).
The digitalization of equipment and the increased capacity of
storage have led to the creation of bigger data sets that have
grown to reach hundreds of thousands, or even millions of
entries. Besides, it has led to the apparition of large collections of
heterogeneous data, such as images, sounds, or texts, which differ
from tabular data sets by their high dimensionality. For instance,
an X-ray image from the medical imaging scenario is made of
thousands of pixels providing a spatial representation of an organ,
possibly including several color channels (see Figure 2b).
2) Complexity of Models
Models are at the core of machine learning systems, and consist
in transforming input data into predictions using simple
operations. For example, the logistic regression applied to the
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 77
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
