IEEE Consumer Electronics Magazine - July/August 2021 - 85

Pðiteþ1Þ
u;s
¼ PðiteÞ
u;s þ1ð1PðiteÞ
u;s Þ2ð1 ^rðiteÞ
sðiteþ1Þ ¼ sðiteÞ
Pðiteþ1Þ
u;s
¼ PðiteÞ
u;s
þ2ð1 ^rðiteÞ
u;s Þ
1^rðiteÞ
jSj 1
u;s PðiteÞ
u;s
1
PðiteÞ
u;s
; sðiteþ1Þ 6¼ sðiteÞ
(2b)
where1;2 2½0; 1 are the learning parameters,
and based on them, three different LA
approaches are possible: (1) Linear reward penalty
(LRP), where1 ¼2; (2) Linear reward-penalty
(LR-P), where1 >>2; (3) Linear
reward inaction (LRI), where2 ¼ 0. Here, each
TTPM performs a thorough, a less thorough, and
a very limited exploration of its available strategies,
respectively. Following convergence, each
TTPM has selected the most efficient combination
of Slave TTPs s. A detailed comparative evaluation
of the LRP, LR-P, and LRI TTPs selection
algorithms is presented in the " Reinforcement
Learning-Based TTPs Selection " section.
Autonomous Offloading of Fractional
Amounts to Slave TTPs Following the selection
process, TTPM then determines the fraction
fu;t% that it will offload to each selected Slave
TTP t 2 s, with the goal of maximizing performance
and privacy. Regarding the worst case privacy
scenario outlined in the " Transaction
Timelines " section, TTPM validates the issued
tokens and immediately sends requests to the
Slave TTPs to generate blinded tokens, which are
then transmitted to the Bank. Other TTPM will
make requests concurrently, thereby obscuring
Alice's issued tokens among the multiple fractional
requests for other customers. The parallel
processing capability provided by multiple Slave
TTPs also reduces the turn-around time for
Alice's exchange operation.
TTPM's strategy is Fu;i ¼½fu;1
PN
; ...;fu;t
fu;N,
; ...;
t¼1 fu;t ¼ 1, i 2 I ¼f1; ...; jIjg, where I
is the number of offloading combinations. By
selecting strategy Fu;i, TTPM experiences the following
reward:
rðiteÞ0
u;Fu;i
P
¼ P
t2sðfðite0Þ
u02jUt j;t2s ðfðite0Þ
u0;t Mu0
u;t MuFt
Þ
Þ
P
t2s Ft
(3)
July/August 2021
Pðite0þ1Þ
u;Fu;i
¼
maxfe
r^0ðite0Þ
u;F0
:
ðite0Þb
u;i
;e
r^ðite0Þ
u;Fu;i
ðite0Þb
g
85
Pðite0þ1Þ
u;Fu;i
¼
maxfe
r^0ðite0Þ
u;F0
ðite0Þb
u;i
e
r^ðite0Þ
u;Fu;i
;e
ðite0Þb
(4d)
r^
ðite0Þ
u;Fu;i
ðite0Þb
g
e
r^0ðite0Þ
u;F0
ðite0Þb
u;i
(4c)
u;s ÞPðiteÞ
which can be normalized as ^rðite0Þ
u;s ;
(2a)
r
u;Fu;i ¼P
ðite0Þ
u;Fu;i
u2U r
ðite0Þ
u;Fu;i
2
½0; 1 to reflect the reward probability. Four different
artificial intelligent RL algorithms are investigated
to determine Alice's stable offloading
decision, namely: (1) LRI, which belongs to the
category of gradient ascent algorithms; (2) binary
log linear learning (BLLL); (3) max log linear
learning (MLLL); and (4) optimistic Q-learning
with upper bound confidence action selection
(QUC
). The probability of selecting a strategy Fu;i
in iteration ite0 of each algorithm is denoted as
. The probabilistic selection of an offloading
r^ðite0Þ
u;Fu;i
and all offloading
Pðite0Þ
u;Fu;i
strategy considering the LRI gradient ascent algorithm
follows the rule of (2a) and (2b) by utilizing
the reward probability
combinations jIj in (2b).
A benefit provided by the log linear algorithms
is that they enable convergence to the best equilibrium
compared to the LRI algorithm (and in
general the gradient ascent algorithms) by allowing
a more thorough exploration of their strategy
space. In the BLLLandMLLL algorithms, a random
TTPM selects an alternative strategy F0
u;i
normalized reward ^r0ðite0Þ
u;F0
ðite0Þ with
equal probability (while the other TTPM keep the
same actions) and receives the corresponding
ðite0Þ during the explorau;i
tion
phase. In the learning phase, the TTPM
selects different (Fu;i
ðite0þ1Þ ¼ Fu;i
ðite0þ1Þ ¼ F0
(Fu;i
ðite0Þ) offloading strategy based on
u;i
(4a) and (4c), and (4b) and (4d) for the BLLL and
MLLL algorithms, respectively
Pðite0þ1Þ
u;Fu;i
e
¼
e
Pðite0þ1Þ
u;Fu;i
¼
e
r^0ðite0Þ
u;F0
ðite0Þb
u;i
r^
þ e
ðite0Þ
u;Fu;i
ðite0 Þb
r^0ðite0Þ
u;F0
ðite0Þb
u;i
r^
e
þ e
ðite0Þ
u;Fu;i
r^
ðite0Þ
u;Fu;i
ðite0 Þb
(4b)
ðite0Þb
r^0ðite0Þ
u;F0
ðite0Þb
u;i
(4a)
ðite0Þ)orthe same
IEEE Consumer Electronics Magazine - July/August 2021

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - July/August 2021
