IEEE Robotics & Automation Magazine - December 2021 - 25

where al means the value of variable a is updated, which is the
same as variable s, and r is the measured value of
value as rr=
r .t
define the optimal Qr
t r
argmax Qs al
aAl!
"
If we
t (, :)(,) ,
Qs maxaQ sa ,
the determini s t i c policy can be obtained by a =
(, ).
To im plement the PMRA DQN, the
model should embed a multireward mechanism, and the
PER function is applied to the training process. The following
sections will introduce details about the observation, reward,
PER, action, and Q network.
Observation of the Agent Car
On a simulation platform, we can collect data from the firstperson
perspective (the sensor) and the third-person perspective
(the environment). Considering that image and
point cloud data are the most popular input format in autonomous
driving, the agent car is equipped with a front view
camera and a lidar sensor. The red-green-blue front view
camera provides information about the road and surrounding
vehicles, while the single-line lidar sensor measures 2D
depth information in 360º.
Multireward for Driving
We propose an MRA for highway driving to train a better
function approximator to accomplish complex tasks. In the
DQN framework, when the nonlinear function is used to
approximate the Q value function, the update of the Q value
is prone to oscillation, which leads to unstable learning
behavior. For this reason, a target network is introduced. It
can provide a target value, and the online network can provide
an estimate value; the two values can used to obtain the
learning loss. We name the model the MRA DQN. In our
case, the value to approximate is
Q ,r
t
and the approximator is
implemented in the form of a deep neural network. When
training the MRA DQN, the loss function and target value are
formulated as
L ()=-l^h6 yQ(, ;) ,@ii (5)
DQN
yr xQs a
ii E ,, ,sars
DQN
where yi
2
i
i =+ci$ ma (, ;),
ll
the function approximator. Here, i 1i -
sa i
i-1
(6)
DQN is the target value and i is the weight vector for
is encoded by a separated
target network. The meaning of the other notations
remains as previously explained.
The general design idea is to decompose a single learning
task into several subtasks. Subreward functions are proposed
to train different branches of the deep neural network, which
makes the branches optimize toward independent goals. The
original Q network is divided into branches for high-dimensional
feature representation and shares the same feature
map for the low-dimensional network. Therefore, the total
reward Renv
the benefits obtained by various actions:
(, ,)
env
=
!
Rs asll/ (, ,),
kP
Rs ask
(7)
where P is the set of policies. As shown in (7), the subnetworks
are trained with the reward function emphasizing
different aspects. The advantage of this design is that the optimization
of the whole
network is constrained to
a few possible directions
by observable variables
[10], which makes the
model's decision more
interpretable for humans,
and the training burden
is lightened.
Safety and efficiency
The advantage of this
design is that the
are essential factors that
should be taken into account
for highway driving.
We consider four driving
preferences: higher speed,
more overtaking, less lane changing, and no collisions. In
other words,
optimization of the whole
network is constrained to a
few possible directions by
observable variables.
P := {speed changing, overtaking, lane changing,
collision}. The reasonability and formulation of these actions
are as follows.
Reward for Speed Rs
The higher the speed is, the more time is saved. Encouraging
faster driving brings an improvement of efficiency. The speed
reward is closely related to velocity v as
R =
s
()$
-
-
vv
vv r
maxmin
min
v
.
In (8), the basic reward value for the speed is ;rv
(8)
vmax and
vmin are the maximum and minimum velocity in a range of
time, so vmin may not be zero.
Reward for Overtaking Ro
Overtaking helps to maintain high speeds and brings
about more efficiency, so we define the reward for passing
vehicles as
R = '
o
!
r
where r Ro
o
the agentcar overtakesother cars
otherwise,
+
.
Penalty for Lane Changing Rl
Safety must be guaranteed when pursuing efficiency. In case
the reward for overtaking grows enthusiastically high, which
may instigate unnecessary passing and cause accidents, a limitation
on lane changing frequency is reasonable. Therefore,
we define the discouragement for lane changing as
returned from the environment is the sum of all
R = '
l
where r Rl
!
+
.
DECEMBER 2021 * IEEE ROBOTICS & AUTOMATION MAGAZINE *
25
-rl
the agent car changeslane
otherwise,
(10)
(9)
IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021
