Computational Intelligence - May 2017 - 64

Comfortness Index

8

1,800

6

1,600

4

1,400

0

0

50
Time (s)
(a)

0.2

Safety Index

Performance Index

2

0.15

1,200
1,000
800
600

0.1

400

0.05

200

0

0

0

50
Time (s)
(b)
Initial Policy

0

50
Time (s)
(c)

First Policy

100

Final Policy

Figure 11 Comparison of control policies' performance: (a) Comfort index; (b) Safety index;
(c) Backward value function.

units. That is, the computational time
is only determined by the number of
the actor network's hidden layer node.
After the offline training process, the
online implementation process only
needs a low computational resource
which is suitable for this kind of real-
time application.
Remark 2: Note that the human driver
can turn off the intelligent cruise control
system through the human-computer
interaction when he/she is not satisfied
with the controller. Then the human
driver controls the vehicle, meanwhile,
the offline driving data is collected con-
tinuously to train the intelligent cruise
controller offline until the actor-critic
networks are converged again. Then the
drivers can update the controller with a
satisfied driving safety and comfort.
V. Test Results
A. Learning Phase

140

100
Desired
MFOC
LQR
PID

120

90
Velocity (km/h)

Clearance (m)

100
80
60

80

70

40
60

20
0

0

50
Time (s)
(a)

100

50

0

50
Time (s)
(b)

100

Figure 12 Trajectory during car following scenario: (a) Clearance; (b) Velocity.

in Eq. (21) to approach the optimal
control policy in Eq. (16). This is the
classical TD learning method in rein-
forcement learning. We have proved
that the actor network can approach
the optimal control policy Eq. (16) and
the critic network can approach the
solution of the Bellman equation Eq.
(15) based on the policy iteration in
[26]. The two networks are updated
alternately based on the policy iteration
method until the converged weights

64

are achieved. Then, we can obtain the
nearly optimal control policy.
Remark 1: Note that the MFOC
algorithm is trained offline based on the
collected data to obtain the converged
weights of the critic and actor net-
works. Then, the nearly optimal con-
troller obtained by the actor network is
applied to the intelligent cruise system
online. The computational complexity
of online implementation process is
O(n), where n is the number of hidden

IEEE ComputatIonal IntEllIgEnCE magazInE | may 2017

In the experiment, the state varies in
ranges: d ! [0, 140] m, Dv ! [- 15, 15]
m/s, v h ! [5, 33] m/s. When v h 1 5 m/s,
it will alert the human driver to take
over. Besides, the acceleration of host car
is in a range of [- 4, 2] m/s 2 by taking
into consideration both the mechanical
performance of a car and the comfort.
Correspondingly, the action is in a range
of a ! [- 4, 2] m/s 2 .
According to the state space and
Eq. (12): E [a] = 0.5 m/s 2 when p = 0.25;
E [a] =-0.52 m/s 2 when p = 0.42. p
is set as 0.25 if the velocity of the host
car is below 5 m/s, and it will cause the
car to speed up overall. After its velocity
exceeds 35 m/s, p is set as 0.42 to make
the car speed down overall. This data
collecting strategy is applied to the host
car to explore the state space. Figure 8 (a)
and (b) show the velocity and accelera-
tion trajectories of the host car. The
strategy starts when the car is at a veloc-
ity of 35 m/s, and random actions act
on the car. The car speeds down overall
when the velocity reaches the upper
limit. Otherwise, it speeds up. Figure 8 (c)
is the distribution of the velocity and
the acceleration. It indicates the state
points are approximate uniformly dis-
tribute in the space.



Table of Contents for the Digital Edition of Computational Intelligence - May 2017

Computational Intelligence - May 2017 - Cover1
Computational Intelligence - May 2017 - Cover2
Computational Intelligence - May 2017 - 1
Computational Intelligence - May 2017 - 2
Computational Intelligence - May 2017 - 3
Computational Intelligence - May 2017 - 4
Computational Intelligence - May 2017 - 5
Computational Intelligence - May 2017 - 6
Computational Intelligence - May 2017 - 7
Computational Intelligence - May 2017 - 8
Computational Intelligence - May 2017 - 9
Computational Intelligence - May 2017 - 10
Computational Intelligence - May 2017 - 11
Computational Intelligence - May 2017 - 12
Computational Intelligence - May 2017 - 13
Computational Intelligence - May 2017 - 14
Computational Intelligence - May 2017 - 15
Computational Intelligence - May 2017 - 16
Computational Intelligence - May 2017 - 17
Computational Intelligence - May 2017 - 18
Computational Intelligence - May 2017 - 19
Computational Intelligence - May 2017 - 20
Computational Intelligence - May 2017 - 21
Computational Intelligence - May 2017 - 22
Computational Intelligence - May 2017 - 23
Computational Intelligence - May 2017 - 24
Computational Intelligence - May 2017 - 25
Computational Intelligence - May 2017 - 26
Computational Intelligence - May 2017 - 27
Computational Intelligence - May 2017 - 28
Computational Intelligence - May 2017 - 29
Computational Intelligence - May 2017 - 30
Computational Intelligence - May 2017 - 31
Computational Intelligence - May 2017 - 32
Computational Intelligence - May 2017 - 33
Computational Intelligence - May 2017 - 34
Computational Intelligence - May 2017 - 35
Computational Intelligence - May 2017 - 36
Computational Intelligence - May 2017 - 37
Computational Intelligence - May 2017 - 38
Computational Intelligence - May 2017 - 39
Computational Intelligence - May 2017 - 40
Computational Intelligence - May 2017 - 41
Computational Intelligence - May 2017 - 42
Computational Intelligence - May 2017 - 43
Computational Intelligence - May 2017 - 44
Computational Intelligence - May 2017 - 45
Computational Intelligence - May 2017 - 46
Computational Intelligence - May 2017 - 47
Computational Intelligence - May 2017 - 48
Computational Intelligence - May 2017 - 49
Computational Intelligence - May 2017 - 50
Computational Intelligence - May 2017 - 51
Computational Intelligence - May 2017 - 52
Computational Intelligence - May 2017 - 53
Computational Intelligence - May 2017 - 54
Computational Intelligence - May 2017 - 55
Computational Intelligence - May 2017 - 56
Computational Intelligence - May 2017 - 57
Computational Intelligence - May 2017 - 58
Computational Intelligence - May 2017 - 59
Computational Intelligence - May 2017 - 60
Computational Intelligence - May 2017 - 61
Computational Intelligence - May 2017 - 62
Computational Intelligence - May 2017 - 63
Computational Intelligence - May 2017 - 64
Computational Intelligence - May 2017 - 65
Computational Intelligence - May 2017 - 66
Computational Intelligence - May 2017 - 67
Computational Intelligence - May 2017 - 68
Computational Intelligence - May 2017 - 69
Computational Intelligence - May 2017 - 70
Computational Intelligence - May 2017 - 71
Computational Intelligence - May 2017 - 72
Computational Intelligence - May 2017 - Cover3
Computational Intelligence - May 2017 - Cover4
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202311
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202308
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202305
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202302
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202211
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202208
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202205
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202202
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202111
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202108
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202105
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202102
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202011
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202008
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202005
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_202002
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201911
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201908
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201905
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201902
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201811
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201808
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201805
https://www.nxtbook.com/nxtbooks/ieee/computationalintelligence_201802
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring17
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring16
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring15
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring14
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_summer13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_spring13
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_winter12
https://www.nxtbook.com/nxtbooks/ieee/computational_intelligence_fall12
https://www.nxtbookmedia.com