IEEE Electrification - March 2021 - 51

communicate via an equivalent SDN. This SDN can be
modeled by an equivalent Markov chain model obtained
by projecting the original Markov model via P4 .
The local control objectives can be achieved via P1 and
P3 . Simplified, reduced-dimensional control inputs j(t)
and m r (k) will need to be designed for the two layers and
finally projected back to their respective full-dimensional
models together with the local controllers via inverse projections K T and U T, as shown in Figure 5. One may use
both PMU and communication channel data to find K and
U such that they not only maintain closed-loop stability
but also guarantee close mimicry to the performance of
the ideal closed loop presented in Figure 4.
The inclusion of the SDN communication layer and
extending the H-RL concept to its Markov chain model
will define a completely new research challenge. The
important issues of mixed continuous-time learning versus discrete-time learning will need to be addressed. RL in
continuous and discrete time, as shown in the works of
Abouheaf et al. 2014 and Vamvoudakis 2017, are quite different. As the SDN bandwidth models are stochastic, one
may also need to extend the H-RL-LQR to stochastic control systems.

Wide-Area Power Grid Model

PMU
Data
y (t )
Projection

Control
Input
u (t )
Λ

ω (t)

Inverse
Projection

ΛT

Learning
Reduced-Order Model
.
x (t ) = Ax (t) + Bω (t)

ϑ (t)

ϑ (t ) = Cx (t) + Dω (t)
Simplified Cloud Layer Model

~
ϑ (t) = K ω (t)
Simplified SDN Layer
Equivalent
Congestion
Service-Rate
Information ξ r i (k ) for
Commands λr (k )
the i th-Equivalent Link
~
~
ξ r (k + 1) = Pξ r (k + 1) + Qλr (k )
Φ

Learning
ΦT
Simplified Markov Model

ξ 1(k + 1)
ξ n(k + 1)

ξ 1(k )
ξ n(k )

λ 1(k )
λ n(k )

Markov Chain Model for Service-Rate Control
Figure 5. H-RL-LQR wide-area control.

How to Generate State Feedback
Note that to apply the H-RL-LQR, we will need full state
availability for our power system models. Accessing all
the states of a large power grid, however, is a daunting
task due to the limited number of PMUs. Therefore, following Gol and Abur 2013, we must assume that a sufficient number of PMUs have been placed in the grid so as
to satisfy the geometric observability of all the synchronous generator buses and, that the admittance matrix of
this set of buses is known to the designer even if the rest
of the model is unknown. Under this assumption, using
the current and voltage measurements from the PMUs,
one can compute the phasors at the generator buses and
track them back to estimate the generator states, as
shown in the work of Singh and Pal 2014. Similarly, one
must also assume that a sufficient number of current
and voltage sensors are placed at the DER and FACTS
buses to measure their dominant states, including their
bus frequencies.

Centralized Versus Distributed Learning
Another important issue is whether the learning phase
for both the power grid and the SDN should be implemented in a centralized or distributed way. Centralized
learning and control is generally preferred in an SDN, but
processing all of the PMU data in one central location to
learn K local and K global may be impractical for the grid.
Therefore, in addition to the control, the learning process
may need to be distributed across various control centers. K local, for example, can be learned at individual arealevel control centers using local PMU data while K global
can be learned at the independent system operator using
averaged PMU data from each area. These architectural
issues need to be formalized in parallel to the wide-area
control design.

Conclusion
This article serves as a technical invitation to engineers
for entering the challenging and attractive research field
of machine learning for the wide-area monitoring and
control of power systems. Several new research directions pertaining to model-free wide-area control were
presented in this article, and a strong dependence of the
control on various properties attributed to learning
architectures was established. Evidently, there are several challenges that need to be surmounted to implement
the designs that were described. This requires a strong
knowledge of machine learning, system identification,
stochastic control, robust control, optimization, and
related topics in signal processing. It is envisioned that
this topic will be viewed by budding and established
control theorists as a challenging and attractive opportunity. We hope that the compelling societal importance
of harnessing the data revolution in power and energy
systems will also serve as an additional motivation to
enter this endeavor.

IEEE Electrific ation Magazine / MARCH 2 0 2 1

IEEE Electrification - March 2021

Table of Contents for the Digital Edition of IEEE Electrification - March 2021