IEEE Power & Energy Magazine - May/June 2022 - 45

in different building environments. Deep RL, with the powerful
embedded DNN providing more generalization and
adaptability, is an ideal approach for achieving a flexible and
intelligent HVAC control strategy. More details of the deep
RL approach will be discussed in the next sections.
Deep-Q Network for HVAC Control
The deep-Q network (DQN) is a combination of Q-learning
and a DNN. The main idea behind the DQN is to build
a neural network to estimate the optimal control action
in a discrete domain. The neural network functions like a
complex lookup table, with the input being a state and the
output being the action values for all of the possible control
actions at the current state. In the DQN, this action value is
called the Q-value, where a higher value indicates a better
or more effective action. This DQN approach selects the
action with the highest Q-value
evaluated by the neural network.
The DQN approach adopts a
stabilization strategy to train two
neural networks simultaneously,
namely, the target and behavior
networks. The function of the target
network is to provide stablelabeled
samples for the behavior
network to learn. The DQN converges
when the outputs from the
two networks are close to each
other. An overview of the DQN
approach for HVAC control is
shown in Figure 2(a).
Deep Deterministic Policy
Gradient for HVAC Control
The deep deterministic policy gradient
(DDPG) for HVAC control
is specially designed for solving
problems with continuous variables.
In the described DQN control,
the neural network outputs
all of the possible action values,
and the number of action values
generated is limited. As a result,
the algorithm processes discrete
actions. In the case of HVAC control,
given the control action as the
setpoint and an example range for
the setpoint from 20 to 22 °C, we
need to discretize the proposed
range. For instance, if we set the
step size as 1 °C, then the potential
action set becomes {20 °C,
21 °C, 22 °C}, and the algorithm
learns to choose from the three
actions.
In contrast,
may/june 2022
the DDPG
Residential HVAC
System
System State
Setpoints
Environment
Data
Collection
Reply Buffer
  
Minibatch Training
(a)
Actor: Outputs the Actions
DDPG RL Agent
Residential HVAC
System
Soft
System
State
Behavior
Network
Setpoints
Environment
Data
Collection
Minimize Difference
Critic: Evaluates the Actions
Reply Buffer
Minibatch Training
(b)
figure 2. A multizone HVAC control framework with (a) a DQN and (b) a DDPG.
DDPG: deep deterministic policy gradient.
ieee power & energy magazine
45
Maximize
Q-Value
Behavior
Network
Update
Target
Network
Action
Soft
Update
Target
Network
algorithm does not require the action space to be discretized.
Given the setpoint range as [20 °C, 22 °C], the algorithm
can generate one continuous number within the range. The
ability to handle a continuous action space makes the DDPG
algorithm more suitable for solving problems where the control
action is a continuous variable.
The DDPG algorithm can be regarded as an extension of
the DQN algorithm. In the algorithm, there are two types of
neural networks applied: the actor and critic networks. The
actor network outputs a deterministic control action based
on the given current state. The critic network outputs the
Q-value based on both the state and action provided by the
actor network. The actor network is further updated by maximizing
the Q-value under the current policy, and the critic
network is updated by minimizing the mean square error
of the Q-value, which is the same as the DQN algorithm. In
DQN RL Agent
Update
Behavior
Network
L = DQN Loss
Current Q-Values Target Q-Values
Soft
Behavior
Network
Update Target
Network
IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022
