IEEE Power & Energy Magazine - May/June 2022 - 53

The results signify that the deep RL approach is of considerable
potential for online applications in solving complex control and
optimization problems like residential demand management.
verifies the accuracy of the fine-tuned building simulation
model. In addition, the bottom plot of Figure 8(b) shows
the power consumption from the baseline case (in blue) and
measured power (in red); the difference between the baseline
power consumption and measured power in Figure 8(b)
is greater than that in Figure 8(a), which demonstrates the
considerable difference between the fixed-setpoint baseline
case and " simulated DQN " case.
Table 4 shows a breakdown of the daily energy cost and
consumption comparisons for the DQN-based RL control
approach and fixed-setpoint baseline case. The daily cost
savings from the DQN control (either simulated or deployed)
range from 0 to 28% (except for a couple of outlier days with
negative cost savings), depending on the day. While there are
some fluctuations in the day-to-day comparison of energy
use as well as the cost of the simulated and measured DQN
cases, the overall energy use and cost over the 11-day deployment
have a difference of less than 3%. This indicates that the
building simulation is well calibrated. The significant observation
from Table 4 is that the DQN cases (either simulated
or deployed) consumed more energy than the baseline while
still managing to reduce the total cost by 11.7% and 12.8%,
respectively, in comparison with the baseline case. This is
because the DQN control preheats the home to a higher temperature
during low-price periods such that the total cost is
decreased, while more energy is consumed, and the comfort
level is better.
Note that the minor difference between the simulated
DQN case and DQN deployment is likely due to a combination
of factors, including small inaccuracies associated
with the building model compared to the real-world building
response and difference in HVAC response over the decision
interval of 5 min. Future performance improvements during
deployment could be achieved by decreasing the decision
interval of the DQN control to a shorter interval and the
inclusion of online learning to fine-tune the decisions of the
DQN control over a longer operational period.
Summary
This article explores the application of deep RL approaches
to implement energy management in a multizone residential
HVAC system to minimize energy costs and maintain user
comfort. Both simulation and real-world deployment results
demonstrate that the deep RL approach can learn an HVAC
control strategy that is more economical, generalized, and
adaptive than either the rule-based or simple fixed-setpoint
control strategy. The results signify that the deep RL
may/june 2022
approach is of considerable potential for online applications
in solving complex control and optimization problems like
residential demand management.
For future directions, an interesting topic would be to
further investigate the generalization ability of deep RL
approaches, for example, to make the control workable for
various scenarios, including both cooling and heating as
well as idle scenarios, without additional retraining efforts.
Another promising direction would be physics-informed
deep RL, which would introduce physical laws to guide the
exploration of the approach and further improve learning
efficiency. Further, similar approaches based on machine
learning may be extended from HVAC control to other
building energy controls.
For Further Reading
V. Mnih et al., " Human-level control through deep reinforcement
learning, " Nature, vol. 518, no. 7540, pp. 529-533,
2015, doi: 10.1038/nature14236.
T. P. Lillicrap et al., " Continuous control with deep reinforcement
learning, " 2015, arXiv:1509.02971.
Y. Du et al., " Intelligent multi-zone residential HVAC
control strategy based on deep reinforcement learning, "
Appl. Energy, vol. 281, p. 116,117, Jan. 2021, doi: 10.1016/j.
apenergy.2020.116117.
Y. Du et al., " Multi-task deep reinforcement learning
for intelligent multi-zone residential HVAC control, " Electric
Power Syst. Res., vol. 192, p. 106,959, Mar. 2021, doi:
10.1016/j.epsr.2020.106959.
K. Kurte et al., " Evaluating the adaptability of reinforcement
learning based HVAC control for residential houses, "
Sustainability, vol. 12, no. 18, p. 7727, Sep. 2020, doi:
10.3390/su12187727.
Biographies
Yan Du is with the University of Tennessee, Knoxville, Tennessee,
37996, USA.
Fangxing Li is with the University of Tennessee, Knoxville,
Tennessee, 37996, USA.
Kuldeep Kurte is with Oak Ridge National Laboratory,
Oak Ridge, Tennessee, 37830, USA.
Jeffrey Munk is with Oak Ridge National Laboratory,
Oak Ridge, Tennessee, 37830, USA.
Helia Zandi is with Oak Ridge National Laboratory, Oak
Ridge, Tennessee, 37830, USA.
p&e
ieee power & energy magazine
53
IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022
