IEEE Computational Intelligence Magazine - February 2022 - 53

of interpretability, therefore, an ill-defined formulation of Eui
can yield misleading results. As a case in point, an experiment
with P 0=x
in Eq. 3 using ML-100K was run. In this scenario,
MEP is always equal to one; thus, any recommendation is
considered fully explainable.
2) Model Fidelity
Model Fidelity belongs to the second category of metrics. It
aims at evaluating explainability via the proxy. More specifically,
it measures how accurate the interpretable proxy is in generating
an 'explainable version' of the predictions of the black box.
Model fidelity is a typical measure adopted in post-hoc explanation
approaches of black box models [26]. Model fidelity can
be defined as:
Fidelity =
||
||
LProxPred
L
+
,
where L is the set of recommendations generated by the
black box model, and ProxPred is the set of proxy predictions.
Model Fidelity varies from 0 to 1, with 1 being a
perfect match between the black box model and the
interpretable model. Fig. 4 shows the results of the fidelity
of a proxy trained using the recommendations generated
by several black box algorithms
that previously went
through a precision-oriented tuning process (using the
ML-1M dataset).
3) Explainability Score
The fidelity metric considers the number of interpretable recommendations
that can be retrieved w.r.t. the original recommendations.
A different metric is required when one wants to
measure to what extent recommendations generated by a black
box algorithm can be explained using latent factors. To this end,
a new metric has to be defined. This new metric measures the
explainability of a recommended item in terms of the number
of user interactions that can support its explanation. This metric
is called Explainability Score.
More formally, given a user u and a recommendation i, the
Explainability Score of u w.r.t. i is defined as follows:
Expl =
ui
where Iu
||
||
IK
u
ul
+
I
ter containing i (with {, ,, }KK Kn12
,
is the set of interactions of user u, and Kl
(4)
is the clusThen,
the explainability score of a recommendation list Lu
defined as follows:
Expl =Lu || / Explui
L
1
u iL6 ! u
Notice that the set of learned embeddings Kl
used in Eq. 4
can be replaced by a set of item embeddings computed using a
different learning method than clustering. For instance, one can
be interested in considering the most similar item embeddings
w.r.t. a given item recommendation, such as in the case of nearest
neighbours.
FIGURE 4 Model Fidelity when increasing the number D of predictions
of a proxy model for the ML-1M dataset. The fidelity values of the ARs
and kNN proxies are low when only a few predictions are considered.
The proxies are better approximations of the black box when more
predictions are considered. However, this gain in fidelity penalises the
interpretability of the proxies since their complexity increases as well
(e.g., high number of rules or of neighbours). (a) Association Rules
(ARs); (b) Nearest neighbours (kNN)
.
f representing clusters).
is
(5)
4) Other Metrics
recoXplainer also implements precision-oriented metrics,
namely Hit Rate and nDCG, to assess the performance of
black box recommenders.
IV. recoXplainer in Practice
Fig. 5 shows the UML diagram of the main classes implemented
in recoXplainer. The classes can be organised in
five main categories according to the functionalities available
in the library: data handling, model training, recommendation,
explanation, and evaluation.
A. Data Handling
The DataReader class provides functionalities to manipulate
a given dataset, and to convert it to a tensor representation
(for GPU computation). ML libraries such as Pytorch and
Tensorflow already implement pre-processing utilities, e.g., the
DataSet class in Tensorflow, or the DataLoader in Pytorch.
Nonetheless, they do not provide utilities for recommender
0.18
0.20
0.22
0.24
0.26
0.16
0.14
10
0.28
0.30
0.32
0.18
0.20
0.22
0.24
0.26
0.16
10
15 20 25 30
D
(b)
35 40 45 50
15 20 25 30
D
(a)
Algorithm
ALS
BPR
MLP
GMF
35 40 45 50
Algorithm
ALS
BPR
MLP
GMF
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 53
Fidelity
Fidelity
IEEE Computational Intelligence Magazine - February 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
