IEEE Power & Energy Magazine - May/June 2022 - 31

regarding the " outstanding " performance of newly proposed
models. From a practitioner's perspective, such results are
meaningless. The cost of implementing a new predictive
model may be high, but its performance may be mediocre.
Another problem is the limited size and scope of the test
data set. Results are usually reported for just one market
and often comprise a selection of only four weeks. This
favors selectively picking results and leads to inflated forecasting
accuracy.
The last issue is data contamination. The test data set is
not always selected as the last section of the full data set, nor
is the test data set completely independent of the validation/
training data sets. Moreover, model hyperparameters are
sometimes estimated in the test data set.
The Ugly-Handle With Care
The use of evaluation metrics is not standardized for the
price forecasting community. New methods are often compared
based on a single accuracy metric without considering
its properties. Also, statistical testing is not always used to
measure whether the differences in accuracy are statistically
significant. This can lead to selectively picked results and
biased conclusions.
Another issue is the lack of details when proposing
new methods. In particular, the split between training,
validation, and testing is often not reported nor are the
optimal input features of the model or optimal model
hyperparameters. This hinders reproducibility. Moreover,
the computational costs of new methods are often ignored.
As short-term EPF often requires real-time forecasting,
the computational cost is an important metric that should
be considered.
Finally, models are not always recalibrated daily. This
is particularly true for benchmarks. Often, a new proposed
model would be frequently recalibrated, but the benchmark
is estimated only once and directly evaluated for the whole
data set. This leads to reporting worse accuracy and, when
done for benchmarks, leads to artificial accuracy improvements
of new models.
The Good-Best Practices
Based on the issues listed, new work in EPF should aim
to follow a series of good practices to ensure reproducibility
and meaningful comparisons. In particular, it should
ensure that
✔ the test data set comprises at least a year of data and is
based on multiple markets
✔ the test data set is selected as the last section of the
available electricity price series, and hyperparameters
are estimated using a validation data set that is different
from the test data set
✔ any new model is tested against well-known, stateof-the-art,
preferably open source models and wellknown
open access data sets
may/june 2022
✔ to evaluate the models, several metrics are considered,
and one of them is the relative MAE
✔ statistical testing is used to assess whether differences
in the predictive performance are significant
✔ the split and dates of the data set are explicitly stated,
and all inputs and parameters of the model are explicitly
defined
✔ the computational cost of new methods is evaluated
and compared against that of existing approaches
✔ forecasting models are recalibrated daily, not simply
estimated once, and evaluated in the full test
data set.
Acknowledgment
Jesus Lago contributed to this work as an outside activity
and not as part of his role at Amazon.
For Further Reading
R. J. Hyndman and G. Athanasopoulos, Forecasting: Principles
and Practice, 3rd ed. Melbourne, Australia: OTexts,
2021.
G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction
to Statistical Learning, 2nd ed. New York, NY,
USA: Springer Science & Business Media, 2021.
J. Lago, G. Marcjasz, B. De Schutter, and R. Weron,
" Forecasting day-ahead electricity prices: A review of
state-of-the-art algorithms, best practices and an open-access
benchmark, " Appl. Energy, vol. 293, p. 116,983, Jul.
2021, doi: 10.1016/j.apenergy.2021.116983.
R. Weron, " Electricity price forecasting: A review of
the state-of-the-art with a look into the future, " Int. J. Forecasting,
vol. 30, no. 4, pp. 1030-1081, 2014, doi: 10.1016/j.ij
forecast.2014.08.008.
F. Ziel, " Forecasting electricity spot prices using LASSO:
On capturing the autoregressive intraday structure, " IEEE
Trans. Power Syst., vol. 31, no. 6, pp. 4977-4987, 2016, doi:
10.1109/TPWRS.2016.2521545.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning.
Cambridge, MA, USA: MIT Press, 2016. http://deep
learningbook.org
Biographies
Arkadiusz Je˛drzejewski is with the Wrocław University of
Science and Technology, Wrocław, 50-370, Poland.
Jesus Lago is with Amazon, Amsterdam, 1097DP,
The Netherlands.
Grzegorz Marcjasz is with the Wrocław University of
Science and Technology, Wrocław, 50-370, Poland.
Rafał Weron is with the Wrocław University of Science
and Technology, Wrocław, 50-370, Poland, and DB Energy,
Wrocław, 50-541, Poland.
p&e
ieee power & energy magazine
31
http://deeplearningbook.org http://deeplearningbook.org
IEEE Power & Energy Magazine - May/June 2022

Table of Contents for the Digital Edition of IEEE Power & Energy Magazine - May/June 2022
