IEEE Power & Energy Magazine - May/June 2022 - 50

violations. Thus, we may conclude that the DDPG control
can effectively solve unseen physical environments and provide
an efficient and flexible HVAC control strategy after
its offline training.
From these simulation studies, both the DQN and
DDPG controls demonstrate a better HVAC control performance
compared with conventional approaches, like a
fixed-setpoint or simple rule-based control strategy, which
implies their considerable potential for online deployment.
As shown in Table 3, the DDPG-based RL algorithm
shows energy cost savings in the range of 19-29% (average
= 25.9%) compared to the fixed-setpoint control in
the simulation.
Since the DQN-based RL approach achieves slightly
higher energy savings (e.g., >32% energy cost savings) compared
to the fixed-setpoint control, we selected the DQNbased
RL algorithm as the HVAC control strategy to deploy
in the real house. The detailed deployment process and
experiment results are introduced in the next section.
table 3. A comparison of results for different control strategies
DDPG
Rule-Based
Building
Index
1
2
3
4
5
6
7
8
9
10
Cost (US$)
42.22
44.13
52.14
59.66
45.84
42.49
37.47
61.21
35.34
43.19
Comfort
Violation
(min)
31
41
45
101
41
39
24
81
25
59
Cost (US$)
27.78
29.22
36.51
43.94
31.3
27.68
23.51
45.42
21.98
28.41
Comfort
Violation
(min)
1,296
1,586
2,347
3,364
1,879
1,398
1,012
3,520
818
1,323
Cost
(US$)
57.98
60.13
68.52
75.61
62.91
59.06
53.42
76.44
49.9
58.46
Deployment of a Deep RL-Based
HVAC Control Strategy
To further validate the control performance of the deep RL
approach in real-world scenarios, the pretrained DQN control
for controlling the multizone HVAC system was deployed at
the Yarnell Station research house in Knoxville, Tennessee,
as shown in Figure 7. The house
is equipped with a two-stage heat
pump, two-zone control system,
and two smart thermostats. The
two-story house is zoned by floor
with a smart thermostat located
centrally on each floor. Supply air
dampers are used to control the
delivery of conditioned air from
the heat pump to the appropriate
zone(s) based on the call for conditioning
from the thermostats.
The zone controller manages the
Fixed Setpoint
Comfort
Violation
(min)
staging of the heat pump and indoor
airflow rate, which is adjusted based
on the number of zones calling for
conditioning and the " size " settings
of those zones (set using jumpers
on the control board during the
commissioning of the system).
The staging of the heat pump is
controlled based on a supply air
(a)
(b)
(c)
figure 7. The Yarnell Station research house in Knoxville, Tennessee, United States: the
(a) front view, (b) back view, and (c) data acquisition system. (Source: Oak Ridge National
Laboratory, https://www.ornl.gov/content/smart-buildings; used with permission.)
50
ieee power & energy magazine
temperature sensor located in the
duct downstream of the air handler.
The staging is controlled to
maintain a heating mode supply
air temperature of at least 32.2 °C
(90 °F). The power consumption
of the heat pump, therefore, is de -
pendent on the outdoor air temperature,
indoor temperature, and
combination of zones calling for
conditioning. During some condition
combinations, it may only
be possible to elicit a single-stage
response from the two-stage heat
pump. With these many different
conditions affecting the achievable
power response, the system is difficult
to accurately model, making
it a challenging real-world application
for the deep RL approach.
may/june 2022

https://www.ornl.gov/content/smart-buildings

IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022