IEEE Robotics & Automation Magazine - December 2021 - 26

Penalty for Collisions Rc
Collisions are the very last thing expected to happen to the
agent vehicle, so we define the punishment for them as
R = '
c
!
-rc
where, again, r Rl
+
.
where wk
Update Mechanism
The first three reward functions are updated in the branches.
Because collision avoidance should be conducted by all the
policies of lane keeping, lane change, and overtaking, it has a
higher priority, which means Rc
are updated. Finally, the reward in Renv
Rs as (, ,)
env
ks ol,
is updated after the branches
is formulated as
ll
"
=+
!
/ Rs as Rc
,,
k(, ,) .
(12)
Specific information about the reward parameters is presented
in the " Experiment and Results " section.
Detailed Implementation of the MRA DQN Model
For the framework of the MRA DQN, the Q value functions
corresponding to the reward functions
subnetworks are denoted by ,Qs
R ,s R ,o and Rl
in the
Q ,o and Q .l The output
action of the car agent is decided by multiple driving
PER
Qs Rs
Prioritized Encoding With the TD Error
The deviation extent from the estimated Q value to the
targe one indicates the importance of the training sample.
Therefore, we apply TD error PMRA
of samples:
S
Qo
Ro
QMRA
Ql Rl
Psay Qs a(, ;) .
,, ,
MRA ii (, ;)
ikik i
ks ol
=! "
/
,
(16)
To
avoid overfitting caused by overbiased sampling, it is necessary
to build a nonzero probability sampling strategy. What
is more, the priority values should possess significant disparity
so that the important data can be sampled enough times.
The sampling probability Si
of sample i is
Shared
Low-Level
Network
Separated
High-Level Network
Branches for MRA
where Pi
Figure 2. The proposed MRA with the decomposed reward
architecture for three branches of the Q network.
Table 1. The kernel parameter configuration.
Name rv vmax
vmin
rl
S = n
i
/ ()a
i=1
()a
Pi
Pi
,
is the priority of sample i fine-tuned with a coefficient
a ! (, ].01 For uniform sampling, a 0= in (17).
ro rc ii c
Value 1 80 km/h 40 km/h 0.25 0.5 20 1/3 0.99
Prioritized Sampling Based on a Sum Tree
The time complexity of storing linear data is O(),n which is
unaffordable for large-scale training. In this article, the solution
is to apply a sum tree to optimize the storage process, by
which the complexity is reduced to O().
logn In a sum tree,
the leaf nodes record the priority, the branch nodes record the
26 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2021
(17)
is the weight assigned to indicate the importance of
every driving preference. The loss function to minimize in
training and the target value are formulated as
L ()
ii
ii (14)
"
=!
E
,,,,
,,
ks ol,
yR sasQ sa
(, ,)
x
sars / yQ (, ;) ,
2
l ^h;
sa
ki ki
E
ki, kk i 1-ll l
=+ci$ ma (, ;).
(15)
The proposed MRA method can improve the driving performance
of the DQN because the learning objective
becomes diverse. The single reward function limits the network
to a single objective task. With the MRA approach, the
network can simultaneously learn multiple objectives, which
is similar to multiagent reinforcement learning. Because different
learning tasks emphasize features representing varying
information, the MRA DQN will comprehensively consider
a driving scenario.
the agentcar collides
otherwise,
(11)
preferences in discrepant aspects, while the driving preferences
are optimized by the branches in the network by the
reward functions. Therefore, the final QMRA
DQN is calculated by combining all the Q value functions:
:
QsawQs a
,,
MRA ii=
! "
(, ;)
/
ks ol,
k
k(, ;),
(13)
in the MRA
to encode the priority
IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021
