IEEE Power & Energy Magazine - May/June 2022 - 44

By optimally designating the setpoint of the HVAC system,
the energy cost can be minimized while keeping the indoor
temperature within the user's comfort range.
The Deep RL Approach
for HVAC Control
Schematic Description of Deep RL
for the HVAC Control Problem
A critical premise for applying deep RL approaches is
that the problem under investigation is a time-sequential
decision-making process. At each time step, the current
state is only related to the previous state, and the optimal
decision can be made based only on the current state
information. This is the case for the optimal control of a
multizone HVAC system. The indoor temperature at
the current state is only related to the parameters at
the previous time interval, and it is not affected by the
indoor temperature at earlier intervals. By optimally designating
the setpoint of the HVAC system, the energy cost
can be minimized while keeping the indoor temperature
within the user's comfort range. Without losing generality,
in the following discussion, we assume that all HVAC
zones need heating. Also, a zone can simply be regarded
as a floor in a house in this study.
When applying deep RL approaches, four essential elements
should be first defined: the state (s), action (a), state
transition probability (p), and reward (r). In the context of
a multizone residential HVAC control problem, the state
includes the following factors:
✔ the time of day
✔ the current indoor temperature
✔ the current outdoor temperature
✔ a 6-h look-ahead outdoor temperature series for
planning
✔ the current retail price
✔ a 6-h look-ahead retail price series
✔ the lower bound of the user comfort level
✔ the maximum retail price within the next 6 h
✔ the length of time to reach the next price peak.
The action is the setting of the HVAC system setpoint.
It can be either discretely or continuously adjusted within
a certain range. The reward is defined as the negative
sum of the energy consumption cost and comfort violation
cost for the control interval, and the comfort violation
cost is calculated based on how many degrees the
indoor temperature deviates from the user comfort level.
The environment is the entire building or house including
the HVAC system.
This process of applying deep RL for HVAC control
Agent
(DNN-Based RL Controller)
Input
State:
Time,
Temperature,
Retail Price, and
User Comfort Level
Reward:
Energy Cost +
Comfort Violation
Cost
Environment:
(Building With HVAC
System)
figure 1. The RL-based approach for HVAC control. DNN: deep neural network.
44
ieee power & energy magazine
Hidden
Layer
Output
is illustrated in Figure 1. Note that we do not define the
state transition probability for the process. The probability
refers to the probability of transitioning to a
specific next state after taking an action at the current
state. If the state transition probability model is known,
the HVAC control problem can be explicitly formulated
and solved analytically. However,
obtaining an accurate state
transition probability model for
the HVAC control problem is
not a trivial task. This is because
the thermal-dynamic model of
buildings with HVAC systems
is related to a variety of parameters,
including resistances and
capacitors from different building
components; weather factors,
such as outdoor temperature and
solar irradiance; and so on.
Therefore, as previously menAction:
(HVAC
Setpoint)
tioned, a model-free approach like
RL is more suitable for solving
the HVAC control problem. Further,
the building models can
vary, and we need a more generalized
and robust HVAC control
approach that can work efficiently
may/june 2022
IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022
