IEEE Consumer Electronics Magazine - July 2018 - 33

Recommendation efficiency is
measured through the number of
recommendation cycles required
to reach a desired product, also
known as average session length.
in Figure 7(a). This measurement suggested that, in the case of
the CCR, 70% of the users maintained the same final recommendation product when they browsed the full set of products.
The remaining 30% of users switched to a different, better
choice when they had the opportunity to browse all of the
products. Furthermore, with the objective of corroborating the
decision effectiveness of the collaborative framework, the
users' behavior during the recommendation process in task 4
was carefully analyzed.
In Figure 7(b), the ratio of individual (in diagonal-filled
lines) and collaborative cycles (in dot-filled lines) is shown
for both sets of users: those who selected the same product
and those who switched to a different product. When users

35

50
40

25
20
15

30

29.3

6.6
11.7

10

14.4

5

5.0

0

The IC
The CCR
The CCR with
with a Target with a Similar
No Similar
(Task 1)
Target (Task 2) Target (Task 3)

The decision accuracy measurement was used to evaluate
the recommendation efficacy. Users were asked to perform
two tasks, 4 and 5. The decision accuracy was measured by
calculating the percentage of participants who switched to a
different, better option when, once they finished the recommendation session, they browsed all alternatives in the data
set. In these experiments, a decision accuracy rate higher in
the CCR (70%) than in the IC (50%) was observed, as shown

10
0

(a)
35

50

30

30

20
15

32.3

11.02

10
5

8.4

20

Benefit (%)

40

25

10
0

0

RECOMMENDATION EFFICACY

20

Benefit (%)

ASL (Cycles)

30

ASL (Cycles)

task 2 to 6.63 cycles for task 3-most likely due to users realizing that targets were dissimilar and, as a result, preferred to
continue interacting individually.
The analysis of variance (ANOVA) statistical method was
applied to analyze whether the difference between the results of
the CCR (tasks 2 and 3) with respect to the IC (task 1) was statistically significant. The ANOVA was applied in three algorithms, k = 3 (IC, CCR with a similar target, and CCR with
dissimilar target) with k - 1 = 2 degrees of freedom. The
ANOVA results indicated that the differences were significant
among the algorithms, and the p-value of 3.236e -6 was
obtained, which was lower than the critical value, a = 0.05.
Additionally, the multiple-significant Bonferroni test [26] was
applied, which signified that the efficiencies of CCR in tasks 2
and 3 were significantly better than the IC in task 1, separately.
A p-value of 2.4e -6 between IC (task 1) and CCR in task 2, and
8.1e -4 between IC (task 1) and CCR in task 3, was obtained.
Figure 6(b) depicts the results of tasks 4 and 5. In these
tasks, ASL also decreased from 32.31 cycles for users who
worked individually (IC algorithm in task 5) to 19.45 cycles
when they collaborated (task 4). Note that the numbers of collaborative cycles and individual cycles are more balanced in
task 4 than in tasks 2 and 3; this is because users likely started
with an idea more or less free of their desired products, but, at
some point, they realized this idea was not clear enough and
decided to collaborate, as is shown in the video recording and
in the observer's notes taken during the test. In Figure 6(b),
only two algorithms are compared; therefore, the t-test measure to evaluate statistically the difference between the IC and
the CCR was applied. T-test results indicated that differences
were significant among them, and a p-value of 2.5e -5
between the CCR in task 4 and the IC in task 5, which was
lower than critical value, a = 0.05, was derived.
Additionally, charts in Figure 6 depict a blue line, which
shows the benefit in percentage of the CCR algorithm over
IC in both scenarios, with and without the target product.
[Note that the percentage of the benefit is expressed as
Benefit (x, y) = ^1 - ^ y/x hh $ 100, where y and x represent the
ASL of the compared algorithm and the baseline, respectively.]
In Figure 6(a), the collaborative recommendation achieved a
benefit over the IC of 42.9% when users had similar target
products; this benefit was 28.2% for users with dissimilar target products. In Figure 6(b), the collaborative recommendation
(task 4) also achieved a benefit over the IC (task 5) of 39.8%.
These results are aligned with the results obtained using the
simulator in previous studies [8].

The IC Without a
Target (Task 5)

The CCR Without a
Target (Task 4)
(b)

ASL Individual Cycles
ASL Collaborative Cycles
The Benefit CCR over the IC
FIGURE 6. The recommendation efficiency results in the live-user
evaluation. (a) Tasks with a target and (b) tasks without a target.

July 2018

^

IEEE Consumer Electronics Magazine

33



Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - July 2018

Contents
IEEE Consumer Electronics Magazine - July 2018 - Cover1
IEEE Consumer Electronics Magazine - July 2018 - Cover2
IEEE Consumer Electronics Magazine - July 2018 - Contents
IEEE Consumer Electronics Magazine - July 2018 - 2
IEEE Consumer Electronics Magazine - July 2018 - 3
IEEE Consumer Electronics Magazine - July 2018 - 4
IEEE Consumer Electronics Magazine - July 2018 - 5
IEEE Consumer Electronics Magazine - July 2018 - 6
IEEE Consumer Electronics Magazine - July 2018 - 7
IEEE Consumer Electronics Magazine - July 2018 - 8
IEEE Consumer Electronics Magazine - July 2018 - 9
IEEE Consumer Electronics Magazine - July 2018 - 10
IEEE Consumer Electronics Magazine - July 2018 - 11
IEEE Consumer Electronics Magazine - July 2018 - 12
IEEE Consumer Electronics Magazine - July 2018 - 13
IEEE Consumer Electronics Magazine - July 2018 - 14
IEEE Consumer Electronics Magazine - July 2018 - 15
IEEE Consumer Electronics Magazine - July 2018 - 16
IEEE Consumer Electronics Magazine - July 2018 - 17
IEEE Consumer Electronics Magazine - July 2018 - 18
IEEE Consumer Electronics Magazine - July 2018 - 19
IEEE Consumer Electronics Magazine - July 2018 - 20
IEEE Consumer Electronics Magazine - July 2018 - 21
IEEE Consumer Electronics Magazine - July 2018 - 22
IEEE Consumer Electronics Magazine - July 2018 - 23
IEEE Consumer Electronics Magazine - July 2018 - 24
IEEE Consumer Electronics Magazine - July 2018 - 25
IEEE Consumer Electronics Magazine - July 2018 - 26
IEEE Consumer Electronics Magazine - July 2018 - 27
IEEE Consumer Electronics Magazine - July 2018 - 28
IEEE Consumer Electronics Magazine - July 2018 - 29
IEEE Consumer Electronics Magazine - July 2018 - 30
IEEE Consumer Electronics Magazine - July 2018 - 31
IEEE Consumer Electronics Magazine - July 2018 - 32
IEEE Consumer Electronics Magazine - July 2018 - 33
IEEE Consumer Electronics Magazine - July 2018 - 34
IEEE Consumer Electronics Magazine - July 2018 - 35
IEEE Consumer Electronics Magazine - July 2018 - 36
IEEE Consumer Electronics Magazine - July 2018 - 37
IEEE Consumer Electronics Magazine - July 2018 - 38
IEEE Consumer Electronics Magazine - July 2018 - 39
IEEE Consumer Electronics Magazine - July 2018 - 40
IEEE Consumer Electronics Magazine - July 2018 - 41
IEEE Consumer Electronics Magazine - July 2018 - 42
IEEE Consumer Electronics Magazine - July 2018 - 43
IEEE Consumer Electronics Magazine - July 2018 - 44
IEEE Consumer Electronics Magazine - July 2018 - 45
IEEE Consumer Electronics Magazine - July 2018 - 46
IEEE Consumer Electronics Magazine - July 2018 - 47
IEEE Consumer Electronics Magazine - July 2018 - 48
IEEE Consumer Electronics Magazine - July 2018 - 49
IEEE Consumer Electronics Magazine - July 2018 - 50
IEEE Consumer Electronics Magazine - July 2018 - 51
IEEE Consumer Electronics Magazine - July 2018 - 52
IEEE Consumer Electronics Magazine - July 2018 - 53
IEEE Consumer Electronics Magazine - July 2018 - 54
IEEE Consumer Electronics Magazine - July 2018 - 55
IEEE Consumer Electronics Magazine - July 2018 - 56
IEEE Consumer Electronics Magazine - July 2018 - 57
IEEE Consumer Electronics Magazine - July 2018 - 58
IEEE Consumer Electronics Magazine - July 2018 - 59
IEEE Consumer Electronics Magazine - July 2018 - 60
IEEE Consumer Electronics Magazine - July 2018 - 61
IEEE Consumer Electronics Magazine - July 2018 - 62
IEEE Consumer Electronics Magazine - July 2018 - 63
IEEE Consumer Electronics Magazine - July 2018 - 64
IEEE Consumer Electronics Magazine - July 2018 - 65
IEEE Consumer Electronics Magazine - July 2018 - 66
IEEE Consumer Electronics Magazine - July 2018 - 67
IEEE Consumer Electronics Magazine - July 2018 - 68
IEEE Consumer Electronics Magazine - July 2018 - 69
IEEE Consumer Electronics Magazine - July 2018 - 70
IEEE Consumer Electronics Magazine - July 2018 - 71
IEEE Consumer Electronics Magazine - July 2018 - 72
IEEE Consumer Electronics Magazine - July 2018 - 73
IEEE Consumer Electronics Magazine - July 2018 - 74
IEEE Consumer Electronics Magazine - July 2018 - 75
IEEE Consumer Electronics Magazine - July 2018 - 76
IEEE Consumer Electronics Magazine - July 2018 - 77
IEEE Consumer Electronics Magazine - July 2018 - 78
IEEE Consumer Electronics Magazine - July 2018 - 79
IEEE Consumer Electronics Magazine - July 2018 - 80
IEEE Consumer Electronics Magazine - July 2018 - 81
IEEE Consumer Electronics Magazine - July 2018 - 82
IEEE Consumer Electronics Magazine - July 2018 - 83
IEEE Consumer Electronics Magazine - July 2018 - 84
IEEE Consumer Electronics Magazine - July 2018 - 85
IEEE Consumer Electronics Magazine - July 2018 - 86
IEEE Consumer Electronics Magazine - July 2018 - 87
IEEE Consumer Electronics Magazine - July 2018 - 88
IEEE Consumer Electronics Magazine - July 2018 - 89
IEEE Consumer Electronics Magazine - July 2018 - 90
IEEE Consumer Electronics Magazine - July 2018 - 91
IEEE Consumer Electronics Magazine - July 2018 - 92
IEEE Consumer Electronics Magazine - July 2018 - 93
IEEE Consumer Electronics Magazine - July 2018 - 94
IEEE Consumer Electronics Magazine - July 2018 - 95
IEEE Consumer Electronics Magazine - July 2018 - 96
IEEE Consumer Electronics Magazine - July 2018 - Cover3
IEEE Consumer Electronics Magazine - July 2018 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20240102
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20231112
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230708
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230506
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230304
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20230102
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20221112
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220708
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220506
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220304
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20220102
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20211112
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210708
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210506
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_20210304
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202010
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202009
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202007
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202004
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202003
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_202001
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201910
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201909
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201907
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201905
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201903
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201901
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201811
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201809
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201807
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201805
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_201803
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_july2017
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_april2017
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_january2017
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_october2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_july2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_april2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_january2016
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_october2015
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_july2015
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_april2015
https://www.nxtbook.com/nxtbooks/ieee/consumerelectronics_january2015
https://www.nxtbookmedia.com