IEEE Power & Energy Magazine - May/June 2022 - 48

We may conclude that the DDPG control can effectively solve
unseen physical environments and provide an efficient and flexible
HVAC control strategy after its offline training.
The DDPG algorithm is trained for 300 episodes. After
the training, the pretrained DDPG algorithm is further validated
in two unseen scenarios: 1) with the Knoxville TMY
data from 1-20 January 2020 and 2) with the same weather
data as the first scenario but with 10 building models that
have different
thermal mass parameters from the simulation
testbed. The DDPG approach is compared with two
Cost Comparison Daywise
8
7
6
5
4
3
2
1
Baseline Cost
RL Cost
benchmark control strategies: 1) a rule-based case, where the
temperature is set at the lowest during the peak price hours
and the highest during the off-peak price hours to achieve the
preheating effect to reduce energy costs, and 2) a fixed-setpoint
case, where the setpoint is always at the highest value
of the setpoint range to avoid violation of user's comfort level.
In the first scenario, the final optimized results of the
Day
US$60.24 US$40.81 (32.25%↓)
RL
Baseline
figure 5. A daywise cost comparison of the DQN-based
HVAC control with the fixed-setpoint baseline for 1-20
December 2019.
table 1. The daily user comfort level.
Time Period
User comfort level:
lower bound (°C)
User comfort level:
upper bound (°C)
0:00-
6:00
18
20
6:00-
12:00
17
19
12:00-
18:00
18
20
18:00-
24:00
19
21
DDPG algorithm and benchmark cases are shown in
Table 2 in which the total energy cost is the accumulated
energy cost over the 10 test days. The average comfort
violation shows the average value, in degrees, by which
the indoor temperature is lower than the setpoint. Table 2
shows that the rule-based case has the lowest total energy
cost because of its temperature setting logic based on
the peak or off-peak price. However, this control strategy
may result in a severe comfort violation because it
always designates the temperature setpoint to the lowest
value at peak price hours. In contrast, since the temperature
is always set at the highest value in the fixed-setpoint
case, there is no temperature violation of the user's comfort
level. Meanwhile, the energy cost is also the highest
among the three control strategies.
Since the temperature is always set at the highest value in
the fixed setpoint case, there is no temperature violation of
user's comfort level. Meanwhile, the energy cost is also the
highest among the three control strategies. The setpoint settings
and the associated indoor temperature variations based
on the three approaches are illustrated in Figure 6. For each
approach, its control strategies for zones 1 and 2 share similar
patterns. Therefore, for the sake of simplicity, we plot
only the control strategies of each approach in zone 1 in the
figure and the control results in the first five days as representative
values.
In all parts of Figure 6, the hour-by-hour yellow recttable
2. A performance comparison of three
HVAC control approaches.
Control Approach
Total energy cost (US$)
Minutes out of user comfort
level
DDPG
55.21
48
Average comfort violation (°C) 0.13
48
ieee power & energy magazine
Rule
Based
39.08
2,617
1.85
Fixed
Setpoint
71.48
angular bars represent the user comfort level as acceptable
temperature ranges, which correspond to Table 1. In Figure
6(a), it can be observed that the DDPG-based control
will designate the setpoint at a relatively low value during
the peak price hours and at a relatively high value during
the off-peak hours. As such, the DDPG-based control can
achieve the preheating effect and reduce energy costs during
the winter.
Figure 6(b) shows the results of the rule-based case, in
which the control strategy designates the setpoint at the
lowest value during peak price hours and the highest value
during off-peak hours. When the outdoor temperature is
extremely low, this control strategy results in severe indoor
may/june 2022
Daily Cost (US$)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022
