IEEE Computational Intelligence Magazine - February 2022 - 53
of interpretability, therefore, an ill-defined formulation of Eui
can yield misleading results. As a case in point, an experiment
with P 0=x
in Eq. 3 using ML-100K was run. In this scenario,
MEP is always equal to one; thus, any recommendation is
considered fully explainable.
2) Model Fidelity
Model Fidelity belongs to the second category of metrics. It
aims at evaluating explainability via the proxy. More specifically,
it measures how accurate the interpretable proxy is in generating
an 'explainable version' of the predictions of the black box.
Model fidelity is a typical measure adopted in post-hoc explanation
approaches of black box models [26]. Model fidelity can
be defined as:
Fidelity =
||
||
LProxPred
L
+
,
where L is the set of recommendations generated by the
black box model, and ProxPred is the set of proxy predictions.
Model Fidelity varies from 0 to 1, with 1 being a
perfect match between the black box model and the
interpretable model. Fig. 4 shows the results of the fidelity
of a proxy trained using the recommendations generated
by several black box algorithms
that previously went
through a precision-oriented tuning process (using the
ML-1M dataset).
3) Explainability Score
The fidelity metric considers the number of interpretable recommendations
that can be retrieved w.r.t. the original recommendations.
A different metric is required when one wants to
measure to what extent recommendations generated by a black
box algorithm can be explained using latent factors. To this end,
a new metric has to be defined. This new metric measures the
explainability of a recommended item in terms of the number
of user interactions that can support its explanation. This metric
is called Explainability Score.
More formally, given a user u and a recommendation i, the
Explainability Score of u w.r.t. i is defined as follows:
Expl =
ui
where Iu
||
||
IK
u
ul
+
I
ter containing i (with {, ,, }KK Kn12
,
is the set of interactions of user u, and Kl
(4)
is the clusThen,
the explainability score of a recommendation list Lu
defined as follows:
Expl =Lu || / Explui
L
1
u iL6 ! u
Notice that the set of learned embeddings Kl
used in Eq. 4
can be replaced by a set of item embeddings computed using a
different learning method than clustering. For instance, one can
be interested in considering the most similar item embeddings
w.r.t. a given item recommendation, such as in the case of nearest
neighbours.
FIGURE 4 Model Fidelity when increasing the number D of predictions
of a proxy model for the ML-1M dataset. The fidelity values of the ARs
and kNN proxies are low when only a few predictions are considered.
The proxies are better approximations of the black box when more
predictions are considered. However, this gain in fidelity penalises the
interpretability of the proxies since their complexity increases as well
(e.g., high number of rules or of neighbours). (a) Association Rules
(ARs); (b) Nearest neighbours (kNN)
.
f representing clusters).
is
(5)
4) Other Metrics
recoXplainer also implements precision-oriented metrics,
namely Hit Rate and nDCG, to assess the performance of
black box recommenders.
IV. recoXplainer in Practice
Fig. 5 shows the UML diagram of the main classes implemented
in recoXplainer. The classes can be organised in
five main categories according to the functionalities available
in the library: data handling, model training, recommendation,
explanation, and evaluation.
A. Data Handling
The DataReader class provides functionalities to manipulate
a given dataset, and to convert it to a tensor representation
(for GPU computation). ML libraries such as Pytorch and
Tensorflow already implement pre-processing utilities, e.g., the
DataSet class in Tensorflow, or the DataLoader in Pytorch.
Nonetheless, they do not provide utilities for recommender
0.18
0.20
0.22
0.24
0.26
0.16
0.14
10
0.28
0.30
0.32
0.18
0.20
0.22
0.24
0.26
0.16
10
15 20 25 30
D
(b)
35 40 45 50
15 20 25 30
D
(a)
Algorithm
ALS
BPR
MLP
GMF
35 40 45 50
Algorithm
ALS
BPR
MLP
GMF
FEBRUARY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 53
Fidelity
Fidelity
IEEE Computational Intelligence Magazine - February 2022
Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - February 2022
Contens
IEEE Computational Intelligence Magazine - February 2022 - Cover1
IEEE Computational Intelligence Magazine - February 2022 - Cover2
IEEE Computational Intelligence Magazine - February 2022 - Contens
IEEE Computational Intelligence Magazine - February 2022 - 2
IEEE Computational Intelligence Magazine - February 2022 - 3
IEEE Computational Intelligence Magazine - February 2022 - 4
IEEE Computational Intelligence Magazine - February 2022 - 5
IEEE Computational Intelligence Magazine - February 2022 - 6
IEEE Computational Intelligence Magazine - February 2022 - 7
IEEE Computational Intelligence Magazine - February 2022 - 8
IEEE Computational Intelligence Magazine - February 2022 - 9
IEEE Computational Intelligence Magazine - February 2022 - 10
IEEE Computational Intelligence Magazine - February 2022 - 11
IEEE Computational Intelligence Magazine - February 2022 - 12
IEEE Computational Intelligence Magazine - February 2022 - 13
IEEE Computational Intelligence Magazine - February 2022 - 14
IEEE Computational Intelligence Magazine - February 2022 - 15
IEEE Computational Intelligence Magazine - February 2022 - 16
IEEE Computational Intelligence Magazine - February 2022 - 17
IEEE Computational Intelligence Magazine - February 2022 - 18
IEEE Computational Intelligence Magazine - February 2022 - 19
IEEE Computational Intelligence Magazine - February 2022 - 20
IEEE Computational Intelligence Magazine - February 2022 - 21
IEEE Computational Intelligence Magazine - February 2022 - 22
IEEE Computational Intelligence Magazine - February 2022 - 23
IEEE Computational Intelligence Magazine - February 2022 - 24
IEEE Computational Intelligence Magazine - February 2022 - 25
IEEE Computational Intelligence Magazine - February 2022 - 26
IEEE Computational Intelligence Magazine - February 2022 - 27
IEEE Computational Intelligence Magazine - February 2022 - 28
IEEE Computational Intelligence Magazine - February 2022 - 29
IEEE Computational Intelligence Magazine - February 2022 - 30
IEEE Computational Intelligence Magazine - February 2022 - 31
IEEE Computational Intelligence Magazine - February 2022 - 32
IEEE Computational Intelligence Magazine - February 2022 - 33
IEEE Computational Intelligence Magazine - February 2022 - 34
IEEE Computational Intelligence Magazine - February 2022 - 35
IEEE Computational Intelligence Magazine - February 2022 - 36
IEEE Computational Intelligence Magazine - February 2022 - 37
IEEE Computational Intelligence Magazine - February 2022 - 38
IEEE Computational Intelligence Magazine - February 2022 - 39
IEEE Computational Intelligence Magazine - February 2022 - 40
IEEE Computational Intelligence Magazine - February 2022 - 41
IEEE Computational Intelligence Magazine - February 2022 - 42
IEEE Computational Intelligence Magazine - February 2022 - 43
IEEE Computational Intelligence Magazine - February 2022 - 44
IEEE Computational Intelligence Magazine - February 2022 - 45
IEEE Computational Intelligence Magazine - February 2022 - 46
IEEE Computational Intelligence Magazine - February 2022 - 47
IEEE Computational Intelligence Magazine - February 2022 - 48
IEEE Computational Intelligence Magazine - February 2022 - 49
IEEE Computational Intelligence Magazine - February 2022 - 50
IEEE Computational Intelligence Magazine - February 2022 - 51
IEEE Computational Intelligence Magazine - February 2022 - 52
IEEE Computational Intelligence Magazine - February 2022 - 53
IEEE Computational Intelligence Magazine - February 2022 - 54
IEEE Computational Intelligence Magazine - February 2022 - 55
IEEE Computational Intelligence Magazine - February 2022 - 56
IEEE Computational Intelligence Magazine - February 2022 - 57
IEEE Computational Intelligence Magazine - February 2022 - 58
IEEE Computational Intelligence Magazine - February 2022 - 59
IEEE Computational Intelligence Magazine - February 2022 - 60
IEEE Computational Intelligence Magazine - February 2022 - 61
IEEE Computational Intelligence Magazine - February 2022 - 62
IEEE Computational Intelligence Magazine - February 2022 - 63
IEEE Computational Intelligence Magazine - February 2022 - 64
IEEE Computational Intelligence Magazine - February 2022 - 65
IEEE Computational Intelligence Magazine - February 2022 - 66
IEEE Computational Intelligence Magazine - February 2022 - 67
IEEE Computational Intelligence Magazine - February 2022 - 68
IEEE Computational Intelligence Magazine - February 2022 - 69
IEEE Computational Intelligence Magazine - February 2022 - 70
IEEE Computational Intelligence Magazine - February 2022 - 71
IEEE Computational Intelligence Magazine - February 2022 - 72
IEEE Computational Intelligence Magazine - February 2022 - 73
IEEE Computational Intelligence Magazine - February 2022 - 74
IEEE Computational Intelligence Magazine - February 2022 - 75
IEEE Computational Intelligence Magazine - February 2022 - 76
IEEE Computational Intelligence Magazine - February 2022 - 77
IEEE Computational Intelligence Magazine - February 2022 - 78
IEEE Computational Intelligence Magazine - February 2022 - 79
IEEE Computational Intelligence Magazine - February 2022 - 80
IEEE Computational Intelligence Magazine - February 2022 - 81
IEEE Computational Intelligence Magazine - February 2022 - 82
IEEE Computational Intelligence Magazine - February 2022 - 83
IEEE Computational Intelligence Magazine - February 2022 - 84
IEEE Computational Intelligence Magazine - February 2022 - 85
IEEE Computational Intelligence Magazine - February 2022 - 86
IEEE Computational Intelligence Magazine - February 2022 - 87
IEEE Computational Intelligence Magazine - February 2022 - 88
IEEE Computational Intelligence Magazine - February 2022 - 89
IEEE Computational Intelligence Magazine - February 2022 - 90
IEEE Computational Intelligence Magazine - February 2022 - 91
IEEE Computational Intelligence Magazine - February 2022 - 92
IEEE Computational Intelligence Magazine - February 2022 - 93
IEEE Computational Intelligence Magazine - February 2022 - 94
IEEE Computational Intelligence Magazine - February 2022 - 95
IEEE Computational Intelligence Magazine - February 2022 - 96
IEEE Computational Intelligence Magazine - February 2022 - 97
IEEE Computational Intelligence Magazine - February 2022 - 98
IEEE Computational Intelligence Magazine - February 2022 - 99
IEEE Computational Intelligence Magazine - February 2022 - 100
IEEE Computational Intelligence Magazine - February 2022 - 101
IEEE Computational Intelligence Magazine - February 2022 - 102
IEEE Computational Intelligence Magazine - February 2022 - 103
IEEE Computational Intelligence Magazine - February 2022 - 104
IEEE Computational Intelligence Magazine - February 2022 - 105
IEEE Computational Intelligence Magazine - February 2022 - 106
IEEE Computational Intelligence Magazine - February 2022 - 107
IEEE Computational Intelligence Magazine - February 2022 - 108
IEEE Computational Intelligence Magazine - February 2022 - 109
IEEE Computational Intelligence Magazine - February 2022 - 110
IEEE Computational Intelligence Magazine - February 2022 - 111
IEEE Computational Intelligence Magazine - February 2022 - 112
IEEE Computational Intelligence Magazine - February 2022 - 113
IEEE Computational Intelligence Magazine - February 2022 - 114
IEEE Computational Intelligence Magazine - February 2022 - 115
IEEE Computational Intelligence Magazine - February 2022 - 116
IEEE Computational Intelligence Magazine - February 2022 - Cover3
IEEE Computational Intelligence Magazine - February 2022 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com