IEEE Systems, Man and Cybernetics Magazine - October 2023 - 59

Proof
The proof is similar to Lemma 1.
■
Theorem 4
Assume that Assumptions 1 and 2 hold and A is stable. If
,, ,,,
QR RR Riij
~
~ and c are selected according to Lemma 2,
*
ij
then, the global consensus tracking error system driven by
the control inputs (8) for the case () ,
~ t 0= or driven by
the control inputs (8) and worst-case disturbance inputs
(9) for the case ()t 0!~
is asymptotically stable.
Proof
From (19), by taking similar steps as in the proof of Theorem
3, one can derive that
ces. Thereby, one can obtain that lim " f = 0.
t
+3
t
N and N are Hurwitz matri()
e
■
Remark
9
For the case where A is stable, [16] developed minmax strategies
for an undisturbed MAS, where the graph weights
were assumed to be small enough to satisfy an inequality
condition (see [16, Assumption 2]). However, in practical
problems, such a condition may not be satisfied for given
graphs. On the other hand, it is impossible to determine
whether the inequality condition is met before solving the
related algebraic Riccati equation (ARE). This makes us
uncertain of whether the proposed control design method
is applicable before addressing the problem.
■
Each agent needs to determine c using Lemma 1 (or
*
2), which requests the agent to know A, B, and C (or B and
C). In addition, every agent is required to solve (16) [or
(19)] to contrive the optimal controller (8), which requires
the agent to know A, B, and C (or A). However, in practical
problems, the system's dynamics are usually unavailable.
Model-Free PI Algorithm
In this section, a model-free PI algorithm is established
for each following agent to solve (16) and obtain the
minmax strategy, as presented in Algorithm 1, where
none of the matrices A, B, and C need to be known in
advance, but (( ), (),( ))xtut tii i
~
can be measured by individual
i. Such an algorithm embodies three parts. In the
first part, inspired by the data-driven off-policy RL algor
ithm [32], each agent
(( ), (),( ))xtut tii i
ii , IN
!
~ measured from the system (1) to identify
B and C. In the second part, each agent employs the parallel
computing scheme to identify A. In the final part, each
agent designs the weight matrices of the performance
function (5), and executes the PI algorithm. The details are
as follows.
First, each agent i identifies B and C. Consider the following
ARE:
AP PA QPBR BP 0
T ** **
{{ {{++ -=1T
ii
i ii
1
where Q 02 and R 0i
i1
i
(20)
2 are set by agent i. A PI algorithm is
reviewed later in this section for solving the ARE (20).
October 2023 IEEE SYSTEMS, MAN, & CYBERNETICS MAGAZINE 59
uses the data
k
k " 3+ ii
=
with
KR .BPii
T
kk=+{{
11
i
Then,
lim PP*{{ and lim KK RB P**
{{k
k " 3+ ii i
==
-1
T { i
.
As the system's dynamics are unknown, for the sake of
solving ARE (20), a data-driven off-policy RL algorithm
[32], [33] is established. For a given bounded signal (),uti
adding and subtracting
{
BK () to (1) yields
{
ixti
k
o ()=+ ++ ~
i
i ii i ii
where
k f== -kk
{
() (()())
{
satisfying that Ai
given by
xt tP xt tx tP xt
xK Ru Kx
T ++ - T
tt
i DDi
+D
i
=+
+- +
#
t
T {{ {kk k
2 x~ xx i1
xPCx QK RK xd
2
kk
kk
i
i
i
Txx x
T
+1T
i
i
ii
i
i
i
i
i
i
T
i
i
i
() () () ()
(( )( () ())
() () ()() ()).
{{
{{
i xx
(24)
Algorithm 1: A Model-Free PI Algorithm
for the General Case.
1: Initialization. Choose the behavior policy as
()
ut Kx ttii ii
=- {
i ().t
QR RRii i
1 ,, ~,,
the initial stabilizing gain Ki{ 0
p
2: Compute ,,x
i
CC and C~
u
i
{{
PK and CP . After calculations,
ii
01
,,
T { 0
i
!
i
B and C are obtained.
5: Solve (29) to attain A.
6: end if
7: Obtain
RR~
8: Choose
9: for
,, and c)
Remarks 6 and 7.
,
L 0i
cG)
,,,
i #m ().Lmin
k =012 f do
= the stabilizing gain Ki
+
PA AP QK c RK
L
kk kk kk
ii
++ +
-=
T
i ii i
1
T c
)2
i c RLi
k~ k
)
i
=- +
11: Update control gains
T
12: if
PP ,ii
1
kk==i
13:
set ,,PP K
))
ii c K1
)
14: break
15: end if
16: end for
17: Use Lemma 1 to get Qi.
18: Output:
i
- #e then
kk+1
i
with AA BK CLii i
kk k
Kc ,( ).==)RB PL
kk k~ k
+- +- T
ii
11 1
ii
c
)2
and L) =i c Li
)
c
i
)2
k+1
.
ci RC Pi
1
)
.
T 1
)
i
i
according to Lemma 1 and
and
10: Solve the following Lyapunov equation for Pi
k
.
3: if the rank condition (27) holds then
4: Solve (26) to get
i
() + p () in a time interval [, ]tti
s
i
i
with
and the exploration signal
Set k 0 .= Let e2 0 be a small constant. Select
ij and ,.~ NRjij
,, ,, is the initial gain
is a Hurwitz matrix.
ii
and Ki
xt Ax tB ut Kx tC tkk ()
01 AA BK
(23)
Using (20)-(23), the ith off-policy Bellman equation is
(22)
Lemma 3
Let K Ri
{ 0
!
matrix [23]. For k 01 2 f= ,, ,, P 0i
kk kk kk
i1
T
i
ii
i
i
i
mn be any initial stabilizing feedback gain
k 2 is the solution for the
#
{
following equation:
() ()ABKP PA BK QK RK
{{ {{ {{
-+ -=-- T
i
(21)
ABCP KL Qii ii
)) )
,, ,, ,, .
IEEE Systems, Man and Cybernetics Magazine - October 2023

Table of Contents for the Digital Edition of IEEE Systems, Man and Cybernetics Magazine - October 2023
