IEEE Computational Intelligence Magazine - February 2022 - 75

the algorithm might be advisable; for high-risk scenarios, an
individual explanation about the decision reached in a specific
case might be necessary (according to recital 71).
Although the definition of personal data is so broad that the
GDPR has been considered the " law of everything " [19], a holistic
regulation of AI is still missing. A proposal for a comprehensive
EU Regulation of AI [9] is currently being discussed based
on the notion of 'Trustworthy AI', which incorporates both
explainability and transparency as mandatory elements among a
list of other important principles such as robustness, security, fairness,
or safety [20]. In this context, the technical notions of
explainability and transparency become crucial requirements to
ensure trust in AI-based decision-making systems.
B. Explainable Artificial Intelligence (XAI)
Making AI systems more understandable first requires making
them transparent, not only regarding the internal implementation
of the models (e.g., parameters), but also with respect to
the full development process (design choices, data uses, testing
procedures, etc.) [21], [22]. Technical transparency alone is not
enough to explain AI models
that are often described as
opaque systems, where relationships between inputs and
outputs are beyond understanding. The concepts of explainability
and interpretability have been introduced to describe how
well a human could understand the decisions of an automated
algorithmic system [7], [23], [24]. Albeit used interchangeably,
explainability is mostly defined as being a subject-centric
notion, whereas interpretability would be an AI model-centric
one, mostly employed in the narrower context of machine
learning models, and thus can be seen as a subset of the broader
notion of explainability [23], [25].
The field of Explainable AI [26], [27] has recently gained a
lot of traction, aiming to provide interpretable elements alongside
predictions. It ranges from a growing literature of technical
work on interpretable models and explainable AI [28]-[30], to
an ongoing discussion about the precise meaning and definition
of explainability and interpretability [6], [25], [31], and to more
procedural questions about the evaluation of existing frameworks
[24]. The meaning of what the technical literature refers
to as explainability of an AI model is very different from the
meaning of an explanation which is generally discussed in other
social contexts (see [7], [23], [24], [31] for the ongoing academic
discussion) or the broader notion of transparency put forward in
similar contexts [20]. In practice, a distinction is made between
approaches aiming at explaining the general mechanisms of a
model (global explanation) and those explaining a decision on a
particular instance (local explanation) [28], in line with the
multi-layered explanation model previously discussed [16].
Machine learning models are typically categorized into interpretable
and non-interpretable models. Interpretable models are
designed to provide reliable and easy-to-understand explanations
of the prediction they output from the start [29], [32]. Their
interpretable nature comes from their simplicity, either by using
only few parameters or operations, or by being conceptually
understandable. Examples include linear and logistic regressions,
decision trees or generalized additive models such as spline fitting.
Non-interpretable models require the use of post-hoc techniques
to generate explanations of decisions. This can be through
the construction of interpretable surrogate models [28], or by
extracting interpretable elements from a range of specific processing
techniques. A classical way to provide insights about
which parts of the data drive a particular behavior is to compute
the contribution of features in the decision-making process, i.e.,
evaluating how much a decision is influenced by specific attributes
of the input data (or features). For some interpretable models,
this could be straightforwardly derived from the model
weights, but dedicated post-hoc approaches have been proposed
to evaluate the significance of data features in a more robust and
systematic way, also applicable to more opaque models. This
includes techniques such as stochastic permutation methods,
local surrogate approximations like LIME [33], or the SHAP values
[34], a popular technique using game-theoretic arguments to
identify the effect of a single feature on an individual entry in
comparison to the average model output computed on training
data. Their use though is often limited to relatively low-dimensional
problems, as they require a minimum computation capability
that is not reachable for complex models. Explaining deep
learning models requires specific techniques [27] that are capable
of handling the high dimensionality of data, and the high number
of parameters typically present in those models. These techniques
should be often adapted to the type of data involved. For
instance, image-based models often rely on visual maps over the
image to highlight important patterns [35].
A major caveat of current approaches is the lack of interactivity
of explanations, that cannot take into account the diversity
of contexts [7] or adapt to the technical background of the
recipient. However, research into explainable AI is very active
and ongoing developments [28], [36]-[41] can be constantly
observed, with some expectations that such limitations might be
successfully addressed. Current EU legislation, and specifically
the above-discussed GDPR provisions, have had a positive
impact on the field of explainable AI, leading to an increasing
number of publications specifically aiming at providing GDPRcompliant
explanations [39], [42]-[44]. These provisions also
introduce additional challenges for research, as the GDPR was
mostly thought for 'tabular data' processing. For complex AI systems,
the implementation of general safeguards (explanation,
contestation, human-in-the-loop) appears more problematic.
III. Challenges of AI for explanations:
insights through high-risk use cases
In this section, the broad technical challenges that question the
feasibility of AI-based algorithmic explanations are explored and
discussed with respect to legal requirements, by means of two use
cases on credit scoring and medical imaging. As illustrated in legal
discussions around the GDPR [2], both cases are prominent highrisk
scenarios for which automated decision-making can have significant
and immediate negative impacts on individual rights.
For each use case, one or several decision-making systems
have been built based on machine learning techniques widely
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 75
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
