IEEE Electrification - December 2022 - 27

actions, the volt-var controller receives a numerical
reward, and the distribution system environment transitions
into a new state.
The action of the volt-var controller agent is defined as
changing the set points of voltage regulating devices and
BTM resources to new levels. The state of the distribution
system environment could include nodal real and
reactive power injections, nodal voltage magnitudes, the
current settings of voltage regulating devices and BTM
resources, and the global time. The reward function is
often defined as the negative of the operational cost,
which includes the cost of power losses, device switching
costs, and voltage constraints violation costs. The
goal of the volt-var controller agent is to learn the optimal
policy, which yields the highest sum of discounted
rewards within the control horizon. The policy learned
by the volt-var controller agent maps states to probabilities
of selecting each possible control action. There are
two value functions associated with a policy, called the
state value function and action value function, which estimate
how good it is for the agent to be in a given state
and to take a given action in a given state and follow the
policy thereafter.
Once the VVC problem is formulated as an MDP, it can
be tackled with a wide range of RL algorithms. The RL
algorithms can be divided into three groups: value-based
methods, policy-based methods, and actor-critic methods.
In value-based methods, the RL agent tries to learn
the state and action value functions and use them to
make control decisions. In policy-based methods, the RL
agent approximates the optimal policy directly without
the need to learn the value functions. In actor-critic
methods, the actor tries to update the learned policy,
while the critic tries to improve the estimates of the
value functions.
Although RL has been successfully demonstrated in
many complex sequential decision-making problems
(e.g., the game of Go), there are many challenges to
deploying it to control real-world
distribution systems with BTM
resources. Some of these challenges
and their proposed solutions
include the following:
x Sample efficiency of RL algorithms:
It is expensive to allow
the RL-based volt-var controller
to interact with the real-world
power distribution system. An
RL-based volt-var controller
requires a large amount of
training data to learn a good
policy. To improve the sample
efficiency of RL-based VVC
algorithms, off-policy RL
algorithms are developed in
favor of on-policy algorithms.
Rt+1
St+1
Off-policy RL algorithms evaluate and improve the target
policy that is different from the behavior policy,
which is used for action selection. In on-policy RL
algorithms, the target policy and the behavior policy
are the same. Thus, off-policy RL algorithms significantly
improve the sample efficiency by allowing the
use of the historical operational data generated from
any controller, which also include the VVC actions
taken by the human operators. To further improve
the sample efficiency of RL-based VVC algorithms,
one could train surrogate models and environment
transition functions to emulate the operations of the
distribution system. Once trained, surrogate models
can be leveraged to generate additional synthetic
training samples.
x Safety of RL algorithms: RL-based volt-var controllers
must be capable of operating the distribution system
in a safe manner even during unforeseen operating
conditions, such as changes in network topology and
BTM resources. The critical operational limits of the
distribution system have to be satisfied all the time. To
improve the safety of RL-based VVC algorithms, many
safe RL algorithms have been developed. One
approach to enforce safety constraints is to formulate
the VVC problem as a constrained MDP by augmenting
the original MDP with a cost function associated
with the operating limits. The goal of the volt-var controller
is to minimize the total operational cost while
ensuring that the expected discounted return with
respect to the cost function is less than a limit. The
other widely used approach is to add a safety layer to
the policy neural network to improve operational constraint
satisfaction.
x Coordination among multiple agents: When the penetration
level of BTM resources is high, the RL-based voltvar
controller needs to properly coordinate the
operations of conventional voltage regulating devices in
the slow timescale, with BTM resources in the fast
State St
Agent
Volt-Var Controller
Reward Rt
Environment: Power Distribution System
OLTC
Voltage
Regulator
Substation
Capacitor
Banks
Solar
PV
Figure 3. VVC through RL. OLTC: on-load tap changer.
IEEE Electrification Magazine / DECEMBER 2022
27
Action At
IEEE Electrification - December 2022

Table of Contents for the Digital Edition of IEEE Electrification - December 2022
