IEEE Electrification - March 2021 - 46

Even if the original
network may not
always be in a
separated form, it
can often be brought
into this form via
appropriate linear
projections.

phenomenon of coherency, which
manifests through low-frequency
interarea oscillations and higher-frequency intra-area oscillations in the
transient response of the power flows
after major faults. The separation
occurs due to a time-scale difference
in the generator dynamics, transmission line lengths, the uneven distribution of renewables across different
zones in the grid, and so on (see
Chow 2013 and Chandra et al. 2016).
Even if the original network may
not always be in a separated form, it
can often be brought into this form
via appropriate linear projections (see
Xue and Chakrabortty 2019). The idea is depicted in Figure 1,
the advantage now being that if one is interested in say,
learning and controlling only the interarea modes, then it
may just suffice to learn a reduced-order model of the
power grid, which would save a tremendous amount of
learning time and numerical complexity. Model-based
designs for such separated control objectives have recently garnered a lot of attention in the control systems community (see Xue and Chakrabortty 2019, Tsubakino et al.
2013, Sadamoto et al. 2018, and Foderaro et al. 2014). Due
to separation, the control goals can be viewed to be driven
by local and global reward functions. The local controllers
in these designs are block decentralized while the global
controllers are usually distributed in the sense of model
reduction or averaging. The model-free or learning version
of these designs, however, is much more challenging as
now both the learner and the controller must follow the
structural constraints imposed on these two layers.
Accordingly, the three components of the hierarchical
reinforcement learning (H-RL)-based scheme discussed in
this article are as follows:
1) To design appropriate projection matrices P that
bring out a possible decomposition of local and

(a)

global control objectives for any
given power system, depending on
its time-and spatial-scale separation, or other similarly patterned
heterogeneities that result in lowrank controllable subspaces
2) To design artificial neural networks
(NNs) that can quickly learn and
predict these projections Pt from
PMU data
3) To design a set of online H-RLbased optimal controllers that
independently serve the local and
global control objectives predicted
by Pt , assuming that the grid
model is either nominally known
or completely unknown. For simplicity, we limit our
study to small-signal, linear time-invariant models
and develop linear quadratic regulator (LQR)-type
wide-area controllers that have been shown to be
quite effective for power oscillation damping (see
Xue and Chakrabortty 2019). Several results on RLbased LQRs have been reported in recent papers,
such as those by Lewis and Vrabie 2009, Vamvoudakis and Lewis 2010, and Jiang and Jiang 2017, which
use adaptive dynamic programming (ADP) and
Kleinman's algorithm. We refer to these benchmark
controllers as centralized RL-based LQRs (C-RL-LQRs).
The H-RL-LQR controller, however, will be significantly more scalable and faster than a C-RL-LQR
due to its hierarchical structure involving smallerdimensional state feedback and parallelization of
the learning loops.
If needed, local controllers can be further decomposed
into sublayers of nested controllers. Too much subclustering, however, may lead to closed-loop instability, and therefore should be considered as a design threshold. The
control gains of the local and global controllers must be
designed in such a way that the overall closed-loop

(b)

Figure 1. (a) A Kron-reduced power system model in the original coordinates and (b) the same power system model in projected coordinates.

I E E E E l e c t r i f i cati o n M agaz ine / MARCH 2021

IEEE Electrification - March 2021

Table of Contents for the Digital Edition of IEEE Electrification - March 2021