IEEE Power & Energy Magazine - May/June 2022 - 47

we repeated the same process for several episodes until the
DQN algorithm converged. Within each episode, at every
control step, the DQN algorithm observes the environment
through various features (called states) and then takes actions
that are implemented in the environment. For example, our
DQN algorithm provides a setpoint as the action, which
leads the environment to evolve into the next state, and then
the same procedure repeats. At the end of each control step,
the environment provides the reward in response to the DQN
algorithm's action. This reward could be the electricity cost
incurred due to the DQN algorithm's action. Based on this
reward, the DQN algorithm then adjusts its Q-table. Readers
may refer to Figure 2 and the relevant description for more
details on DQN algorithm training.
It is important to evaluate the DQN algorithm's training
performance. Throughout the training, we keep track of the
episodic average reward, total electricity cost of operating
HVAC using the DQN for three months, and comfort violations
that count the minutes during which the indoor temperature
violated the user comfort level. The average reward, cost
of operation, and minutes outside of the user comfort level during
the training are shown in Figure 3. In Figure 3(a), the episodic
cumulative reward gradually increases as the training proceeds
and stabilizes in the end. A 23% cost reduction is observed by
the end of the training session in Figure 3(b). The minutes out
of comfort also show a decreasing trend and remain at zero by
the end of the training in Figure 3(c). These observations confirm
the convergence of the DQN-based HAVC control.
The control performance of the pretrained DQN algorithm
is further validated in unseen scenarios: two half-month
Knoxville TMY data sets from 1-20 December 2019 and
11-31 March 2020, which were not used during the training
stage. A fixed-setpoint control strategy is designed as a
baseline case for a comparison with the DQN-based HVAC
control strategy. In the baseline case, the heating setpoint is
always kept at 20 °C.
For brevity, we present the indoor temperature variations
for operating the pretrained DQN-based HVAC controller on
14-15 December in Figure 4. An important observation here
is that the DQN control has learned a preheating strategy.
The DQN control preheated the zones before the peak price.
This was beneficial during the initial hours of the peak period
where the power consumption was zero, as shown by the
power consumption graph in Figure 4. This is how the DQN
control achieved cost savings. The associated daywise energy
cost comparison between the DQN control and baseline cases
for 1-20 December is shown in Figure 5. We observed that
the DQN approach achieved a >32% cost savings over the
fixed-setpoint baseline for both the 1-20 December and 11-31
March data. Full details of the entire duration of 1-20 December
and 11-31 March are not plotted due to space limits.
Training and Validation of the
DDPG for HVAC Control
For the DDPG algorithm, the Knoxville TMY data from
1-30 November 2019 are utilized for training. The control
interval of the DDPG algorithm is 60 min. The state
information input to the DDPG algorithm includes the current
indoor temperature for each zone, outdoor temperature,
and retail price as well as the lower bound of the user comfort
level (Table 1). The DDPG algorithm directly generates a
deterministic, continuous setpoint for each zone of the building's
HVAC system. The range of the setpoint is designated
to be the same as the user comfort temperature zone.
Preheating
Outdoor Temperature
Indoor Temperature (Zone 1)
Indoor Temperature (Zone 2)
RL Action (Zone 1)
20
15
10
5
-5
19,000
December 14
19,500
Minutes
20,000
20,500
December 15
figure 4. The validation of indoor temperature variation for 14 and 15 December with the pretrained DQN model.
may/june 2022
ieee power & energy magazine
47
21,000
RL Action (Zone 2)
Electricity Price
Power Consumption
Preheating
Temperature (°C)

IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022