Instrumentation & Measurement Magazine 24-2 - 120

evader along the ℓ-direction, while K k   13 represents the
control gains which are computed in real-time by the RL
algorithm.
One of the salient features of this setup is its flexibility to
adapt to different problems of various complexities. For instance, for the application in hand, it was found sufficient to
only include the last three tracking errors in Ek. However, it
is possible to add more error signals or remove some without
having to redesign the control algorithm. The same remark
is applicable for modeling the incremental difference in the
surge speed and the orientation angle. In (1a) and (1b), such
signals are shaped as vk1  vk  (ukv  ukv1 ) and  k1   k 
uk ,
respectively. Changing them to some other forms, including
(ukv , ukv1 , ukv2 ,), and (uk , uk1 ,), for example, does not require
the restructure of the algorithm. This exhibits the flexibility
inherited in the learning system where, using only few selfadapted control gains, a filtering behavior can be obtained
during the online learning process.

Temporal Difference Solution
For the RL framework to provide a solution for the pursuer-evader optimization problem, we need to derive the
necessary temporal difference equations (i.e., Bellman
equations) along with the required optimality conditions.
The ultimate objective is to converge to the best policies
ukv* and uk * for each pursuer in order to intercept the evader
ship. Hence, two convex objective functions are considered (each corresponds to the motion in one of the x-y
directions)
	

1  T  



U E ,u
 E Q Ek  R uk
2 k





k


k



 

2







J k   U  E i , ui 	(3)
ik

Taking advantage of the policy expression (2), the optimization process aims at minimizing the performance index J k by
optimizing the gain vector K k. This iterative process may lead
the obtained policy to practically converge towards the optimal policy uk*  arg minu J k , although it may not reach its
k
theoretical value.
A common problem in RL is that J k cannot be calculated.
Motivated by the structure of the cost function U , J k is approximated by a value function V  that is only dependent on two
time-variant signals Ek and uk.

 

	

J k  V  ( E k , uk )	(4)

From (3) and (4), the temporal difference equation is
formed as
	
120	









	




V
( Ek , uk ) U  E k , uk  V  Ek1 , uk1 	(5)

V  ( Ek , uk ) 

E 
1 T T
[ Ek uk ] H   k  ,	
2
 uk 

where H  is a symmetric positive definite matrix with the following block structure
H 
H    EE

 H uE

	



H Eu
	
 
H uu


Note that the structure of the solving value function plays
an important role in how a RL algorithm decides on the best
strategies [14]. With the chosen quadratic structure, the pursuer-evader optimization problem is reduced to finding the
optimal solving value functions, by optimizing H , and consequently computing the optimal control gain K k* using Bellman
optimality principles [11]. To this end, we can conclude that the
optimal policy is
	

* 1
*
) H uE
uk*  arg min V  ( E k , uk )  ( H uu
E k	(6)
uk

Relations (2) and (6) yield the optimal gain
 * 1
*
K *  ( H uu
) H uE
	

	

Applying the optimal policy (6) in (5) yields the following
Bellman optimality equation
	

	



where   { x , y}, 0  Q    33 (positive definite) and 0  R  .
The quality of the applied control strategies is assessed through
a performance measure
	

As with the cost function U , the value function is set as a
quadratic function in the error vector Ek and the control signal uk, such that





*
V
( E k , uk* ) U * E k , uk*  V * ( E k1 , uk* 1 )	(7)

Online Reinforcement Learning
Solution
In most real-world scenarios, it is impossible to find theoretical solutions for (6) and (7). To alleviate this problem, an online
Value Iteration algorithm is employed herein to iteratively approximate such solutions.

Online Value Iteration
Value Iteration is a two-step technique that is used to find the
solutions of many temporal difference forms, such as the Bellman optimality equations (7). In the first step, the solving
value function is updated as
	





 ( r 1)
V
( E k , uk ) U  ( r ) Ek , uk  V  ( r ) ( E k1 , uk1 )	

where r is the value-update index. The second step uses such
update to suggest a new (improved) strategy, given by
	

 1


uk ( r 1)   ( H uu
) H uE


( r 1)

Ek .	

The algorithm cycles through this process until convergence is achieved. The Value Iteration technique is generally
proven to converge. It yields a bounded sequence of non-decreasing solving value functions if the initial solving value

IEEE Instrumentation & Measurement Magazine	

April 2021
Instrumentation & Measurement Magazine 24-2

Table of Contents for the Digital Edition of Instrumentation & Measurement Magazine 24-2
