IEEE Robotics & Automation Magazine - December 2021 - 22

security, and benefit people through high-mobility vehicles.
Decision making is a vital component, as its performance is
closely related to safety. The majority of decision-making
methods in autonomous driving fall into two categories: rule
based and learning based. Rule-based methods explicitly
define restrictions
and apply mathematical
models [1]-[3],
which endows
Autonomous driving is a
promising technology to
improve traffic efficiency,
enhance transportation
security, and benefit
people through highmobility
vehicles.
the
decision-making process
with sufficient
interpretability but
limits performance
on extreme occasions.
Comparatively, learning-based
methods
are usually founded
on supervised learning
and reinforcement
learning, and
they rely on large-scale
data to extract features
automatically, enabling the models to deal with diverse scenarios
with higher stability. Recently, several end-to-end
learning-based methods were proposed [4], [5]. A supervised
learning method directly maps raw input from sensors to
continuous driving actions (the steering wheel angle and driving
speed). However, it needs to be trained on a large-scale
data set, which is hard to collect, especially for discrete driving
decision-making information.
Reinforcement learning is well suited for tasks without
ground-truth labels and accessible to a large amount of
data and much implicit interconnection among features.
Driving decision making can be addressed with a Markov
process [6], which is solvable by reinforcement learning.
The advantage of reinforcement learning-based driving
decision making is that policies can evolve based on
diverse experiences and various scenarios, even if the
information is from a simulator. Remember that a simulator
can present rare cases that are hard to collect from the
real world. Currently, common sensors for tasks related to
autonomous vehicles output high-dimensional perception
data, such as images and point clouds. Therefore, it is suitable
to combine traditional reinforcement learning methods
with deep learning models for better training. Mnih
et al. were the first to combine them in the field of control,
which should be regarded as a big breakthrough for this
area [7]. Then, several milestones were achieved in the
field of deep reinforcement learning [8]-[10]. In [11] and
[12], a specific type of reinforcement learning, DQN, is
applied to autonomous driving. Min et al. proposed a deep Q
learning-based, high-level driving policy decision-making
method in 2018 [6].
The experience replay mechanism and function approximator
are the two elementary parts of the DQN method.
Experience replay can im prove learning efficiency, while a function
approximator with a deep neural network can fit complex
nonlinear data. However, there are two difficulties in this
procedure. First, in traditional experience replay, training
data are often stored without any difference in the memory,
and early data will be removed when the storage is full.
Then, the random sampling method is performed for
experience replay in the training stage, which cannot
demonstrate the learning value of every data sample.
Prioritized Sampling
PMRA DQN
Qs
Rs
Actions
Turn Right
Input
Image
Point Cloud
Ql Rl
Qo
Ro
QMRA
Turn Left
Speed Up
Slow Down
No Action
Figure 1. PER-based deep Q learning with an MRA for highway driving decision making. The PMRA DQN is trained via PER with
decomposition to an MRA for multiobjective tasks. The training data are encoded with TD error and sampled by a sum tree, and the
whole learning task is decomposed into three sublearning ones: higher speed, less lane changing, and more overtaking. The Q network
shares low-level networks but has three branches of the high-level network for the Q value estimation of performance. The final driving
action is determined by combining Q values from the three branches of the Q network.
22 * IEEE ROBOTICS & AUTOMATION MAGAZINE * DECEMBER 2021

IEEE Robotics & Automation Magazine - December 2021

Table of Contents for the Digital Edition of IEEE Robotics & Automation Magazine - December 2021