IEEE Power & Energy Magazine - May/June 2022 - 46

addition, the DDPG algorithm adopts a stabilization strategy
by implementing the behavior and target networks for
both the actor and critic networks, which is similar to the
DQN algorithm. An overview of the DDPG approach for
HVAC control is shown in Figure 2(b).
Evaluation of a Deep RL-Based HVAC
Control Strategy Through Simulation
Our main objective is to evaluate the performance of the deep
RL-based HVAC control strategy in a real-world environment.
However, one cannot deploy an RL algorithm directly
in a real-world environment and allow it to learn from scratch
for two main reasons:
✔ RL-based approaches
are essentially trial-anderror
methods (albeit with high intelligence) in which
-500
-1,000
-1,500
-2,000
-2,500
-3,000
an RL algorithm interacts with the environment (i.e.,
the building or house) and learns from it based on
the reward (i.e., operation cost) from the environment.
Depending on the application, an RL algorithm might
need long experience to learn how to behave optimally.
✔ During the initial learning phase, an RL algorithm tends
to take random actions to explore and understand the
environment. However, homeowners will not be happy
if RL designates random setpoints on their thermostats.
Therefore, instead of directly deploying an RL algorithm
from scratch in a real-world environment, like a
building or house, we train and validate it in a simulation
environment as a starting point, allowing for faster
development and overcoming the challenge of training
RL in a real-world situation. Once we are satisfied with
its performance in the simulation, we can deploy the
pretrained RL model (the trained and deployable RL
algorithm) in a real house.
In this section, we introduce the training and validation
of a deep RL-based multizone residential HVAC control
strategy on a simulation testbed with real-world data. Performance
comparisons with benchmark control strategies
demonstrate the efficiency and generalization ability of
model-free deep RL approaches.
02040
Episode
(a)
380
340
360
320
300
280
260
240
220
200
02040
Episode
(b)
7,000
6,000
5,000
4,000
3,000
2,000
1,000
02040
Episode
(c)
figure 3. The training performance of the DQN algorithm
with the discrete control strategy: the (a) episodic cumulative
reward, (b) cost of operation, and (c) minutes outside the
comfort level.
46
ieee power & energy magazine
60
80
23% Cost
Reduction
60
80
Simulation Setup
The simulated HVAC building model requires weatherrelated
and price data. The weather data are taken from
typical meteorological year (TMY) data from 2019 to 2020
in Knoxville recorded by the National Renewable Energy
Laboratory. For price data, the time-of-use price signals with
a peak price at US$0.25/kWh and an off-peak price at
US$0.05/kWh are applied. These input data sets are applied to
a building simulation software testbed for the following training
and validation of deep RL-based HVAC control strategies.
60
80
Training and Validation of
the DQN for HVAC Control
We first present the simulation result of the DQN algorithm
for multizone residential HVAC control. For training the algorithm,
the Knoxville TMY data from 21 December 2019 to 10
March 2020 were utilized. The simulation step of the HVAC
thermal dynamics is 1 min, and for every 5 min, the algorithm
provides a setpoint control action. The user comfort level is
set to 20-22.22 °C (i.e., 68-72 °F). The state information used
in the DQN control includes all of the elements as listed in the
" Deep RL Approach for HVAC Control " section. The DQN
approach generates a discrete control action by adjusting the
setpoint with a fixed step within the user comfort level.
The DQN algorithm was trained for 75 episodes with
these settings. As mentioned earlier, we used approximately
three months of data from 21 December 2019 to 10 March
2020 for training the DQN algorithm. In this training process,
a single training episode is considered complete after
the DQN algorithm has explored the three-month data. Next,
may/june 2022
Minutes Outside Comfort
Episodic Cumulative Reward
Cost
IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022
