IEEE Computational Intelligence Magazine - May 2022 - 47

When (, ),dp p 0=ts
then the model is perfectly calibrated.
Other measures have also been proposed. Examples include
the expected calibration error and some discretized variants of
the distance from the actual calibration curve to the ideal calibration
curve [16].
VIII. Conclusion
This tutorial covers the design, training and evaluation of
BNNs. While their underlying principle is simple, i.e., just
training an ANN with some probability distribution attached
to its weights, designing efficient algorithms remains very challenging.
Nonetheless, the potential applications of BNNs are
huge. In particular, BNNs constitute a promising paradigm
allowing the application of deep learning in areas where a system
is not allowed to fail to generalize without emitting a
warning. Finally, Bayesian methods can help design new learning
and regularization strategies. Thus, their relevance extends
to traditional point estimate models.
Online resources for the tutorial:
https://github.com/french-paragon/BayesianNeuralNetwork-Tuto
rial-Metarepos
Supplementary material, as well as additional practical examples
for the covered material with the corresponding source code
implementation, have been provided.
Acknowledgments
This material is partially based on research sponsored by the
Australian Research Council https://www.arc.gov.au/ (Grants
DP150100294, DP150104251, and DP210101682), and Air
Force Research Laboratory and DARPA https://afrl.dodlive.
mil/tag/darpa/ under agreement number FA8750-19-2-0501.
References
[1] C. Szegedy et al., " Intriguing properties of neural networks, " 2013, arXiv:1312.6199.
[2] Q. Rao and J. Frtunikj, " Deep learning for self-driving cars: Chances and challenges, "
in Proc. 1st Int. Workshop Softw. Eng. AI Auton. Syst., 2018, pp. 35-38.
[3] J. Ker, L. Wang, J. Rao, and T. Lim, " Deep learning applications in medical image analysis, "
IEEE Access, vol. 6, pp. 9375-9389, 2018, doi: 10.1109/ACCESS.2017.2788044.
[4] R. C. Cavalcante, R. C. Brasileiro, V. L. Souza, J. P. Nobrega, and A. L. Oliveira,
" Computational intelligence and financial markets: A survey and future directions, " Expert
Syst. Appl., vol. 55, pp. 194-211, 2016, doi: 10.1016/j.eswa.2016.02.006.
[5] H. M. D. Kabir, A. Khosravi, M. A. Hosen, and S. Nahavandi, " Neural network-based
uncertainty quantification: A survey of methodologies and applications, " IEEE Access,
vol. 6, pp. 36,218-36,234, 2018, doi: 10.1109/ACCESS.2018.2836917.
[6] A. Etz, Q. F. Gronau, F. Dablander, P. A. Edelsbrunner, and B. Baribault, " How to
become a Bayesian in eight easy steps: An annotated reading list, " Psychonomic Bull. Rev.,
vol. 25, pp. 219-234, 2018, doi: 10.3758/s13423-017-1317-5.
[7] N. G. Polson et al., " Deep learning: A Bayesian perspective, " Bayesian Anal., vol. 12,
no. 4, pp. 1275-1304, 2017, doi: 10.1214/17-BA1082.
[8] J. Lampinen and A. Vehtari, " Bayesian approach for neural networks-Review and
case studies, " Neural Netw., vol. 14, no. 3, pp. 257-274, 2001, doi: 10.1016/S08936080(00)00098-8.
[9]
D. M. Titterington, " Bayesian methods for neural networks and related models, " Statist.
Sci., vol. 19, no. 1, pp. 128-139, Feb. 2004, doi: 10.1214/088342304000000099.
[10] E. Goan and C. Fookes, Bayesian Neural Networks: An Introduction and Survey. Cham:
Springer International Publishing, 2020, pp. 45-87.
[11] H. Wang and D.-Y. Yeung, " A survey on Bayesian deep learning, " ACM Comput.
Surv., vol. 53, no. 5, Sep. 2020, doi: 10.1145/3409383.
[12] A. G. Wilson and P. Izmailov, " Bayesian deep learning and a probabilistic perspective
of generalization, " 2020. [Online]. Available: http://arxiv.org/abs/2002.08791
[13] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA:
MIT Press, 2016. [Online]. Available: http://www.deeplearningbook.org
[14] S. C.-H. Yang, W. K. Vong, R. B. Sojitra, T. Folke, and P. Shafto, " Mitigating belief
projection in explainable artificial intelligence via Bayesian teaching, " Scientific Rep., vol.
11, no. 1, p. 9863, May 2021, doi: 10.1038/s41598-021-89267-4.
[15] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, " On calibration of modern neural
network, " in Proc. 34th Int. Conf. Mach. Learn. (ICML'17), 2017, vol. 70, pp. 1321-1330.
[16] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, " Measuring calibration
in deep learning, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR)
Workshops, Jun. 2019.
[17] D. Hendrycks and K. Gimpel, " A baseline for detecting misclassified and out-ofdistribution
examples in neural networks, " in Proc. 5th Int. Conf. Learn. Representations
(ICLR), Conf. Track Proc., 2017.
[18] Z.-H. Zhou, Ensemble Methods: Foundations and Algorithms, 1st ed. Boca Raton, FL,
USA: CRC, 2012.
[19] F. Galton, " Vox Populi, " Nature, vol. 75, no. 1949, pp. 450-451, Mar. 1907.
[20] L. Breiman, " Bagging predictors, " Mach. Learn., vol. 24, no. 2, pp. 123-140, Aug.
1996, doi: 10.1007/BF00058655.
[21] D. J. C. MacKay, " A practical Bayesian framework for backpropagation networks, "
Neural Comput., vol. 4, no. 3, pp. 448-472, 1992, doi: 10.1162/neco.1992.4.3.448.
[22] P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. Wilson, " What are Bayesian
neural network posteriors really like? " 2021. [Online]. Available: http://arxiv.org/
abs/2104.14421
[23] Y. Gal and Z. Ghahramani, " Bayesian convolutional neural networks with Bernoulli
approximate variational inference, " in Proc. 4th Int. Conf. Learn. Representations (ICLR)
Workshop Track, 2016.
[24] D. P. Kingma and M. Welling, " Stochastic gradient VB and the variational autoencoder, "
in Proc. 2nd Int. Conf. Learn. Representations, ICLR, 2014, vol. 19.
[25] C. Robert, The Bayesian Choice: From Decision-Theoretic Foundations to Computational
Implementation. Springer Science & Business Media, 2007.
[26] J. Mitros and B. M. Namee, " On the validity of Bayesian neural networks for uncertainty
estimation, " in Proc. AICS, 2019.
[27] A. Kristiadi, M. Hein, and P. Hennig, " Being Bayesian, even just a bit, fixes overconfidence
in ReLU networks, " 2020. [Online]. Available: http://arxiv.org/abs/2002.10118
[28] Y. Ovadia et al., " Can you trust your model's uncertainty? evaluating predictive
uncertainty under dataset shift, " in Proc. Adv. Neural Inf. Process. Syst., Curran Associates,
2019, pp. 13,991-14,002.
[29] A. D. Kiureghian and O. Ditlevsen, " Aleatory or epistemic? Does it matter? " Struct.
Saf., vol. 31, no. 2, pp. 105-112, 2009, doi: 10.1016/j.strusafe.2008.06.020.
[30] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft, " Decomposition
of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, " in
Proc. 35th Int. Conf. Mach. Learn., 2018, vol. 80, pp. 1184-1193.
[31] D. H. Wolpert, " The lack of a priori distinctions between learning algorithms, "
Neural Comput., vol. 8, no. 7, pp. 1341-1390, 1996, doi: 10.1162/neco.1996.8.7.1341.
[32] A. Kendall and Y. Gal, " What uncertainties do we need in Bayesian deep learning
for computer vision? " in Proc. 31st Int. Conf. Neural Inf. Process. Syst., (NIPS'17), 2017,
pp. 5580-5590.
[33] T. Auld, A. W. Moore, and S. F. Gull, " Bayesian neural networks for internet traffic
classification, " IEEE Trans. Neural Netw., vol. 18, no. 1, pp. 223-239, Jan. 2007, doi:
10.1109/TNN.2006.883010.
[34] X. Zhang and S. Mahadevan, " Bayesian neural networks for flight trajectory prediction
and safety assessment, " Decis. Support Syst., vol. 131, p. 113,246, 2020.
[35] S. Arangio and F. Bontempi, " Structural health monitoring of a cable-stayed bridge
with Bayesian neural networks, " Struct. Infrastruct. Eng., vol. 11, no. 4, pp. 575-587, 2015,
doi: 10.1080/15732479.2014.951867.
[36] S. M. Bateni, D.-S. Jeng, and B. W. Melville, " Bayesian neural networks for prediction
of equilibrium and time-dependent scour depth around bridge piers, " Adv. Eng.
Softw., vol. 38, no. 2, pp. 102-111, 2007, doi: 10.1016/j.advengsoft.2006.08.004.
[37] X. Zhang, F. Liang, R. Srinivasan, and M. Van Liew, " Estimating uncertainty of
streamflow simulation using Bayesian neural networks, " Water Resources Res., vol. 45, no.
2, 2009, doi: 10.1029/2008WR007030.
[38] A. D. Cobb et al., " An ensemble of bayesian neural networks for exoplanetary atmospheric
retrieval, " Astron. J., vol. 158, no. 1, p. 33, Jun. 2019, doi: 10.3847/1538-3881/
ab2390.
[39] F. Aminian and M. Aminian, " Fault diagnosis of analog circuits using Bayesian neural
networks with wavelet transform as preprocessor, " J. Electron. Testing, vol. 17, no. 1,
pp. 29-36, Feb. 2001.
[40] W. Beker, A. Wołos, S. Szymkuc´, and B. A. Grzybowski, " Minimal-uncertainty prediction
of general drug-likeness based on Bayesian neural networks, " Nature Mach. Intell.,
vol. 2, no. 8, pp. 457-465, Aug. 2020, doi: 10.1038/s42256-020-0209-y.
[41] Y. Gal, R. Islam, and Z. Ghahramani, " Deep Bayesian active learning with image
data, " in Proc. 34th Int. Conf. Mach. Learn., 2017, vol. 70, pp. 1183-1192.
[42] T. Tran, T.-T. Do, I. Reid, and G. Carneiro, " Bayesian generative active deep learning, "
2019. [Online]. Available: http://arxiv.org/abs/1904.11643
[43] M. Opper and O. Winther, " A Bayesian approach to on-line learning, " On-line Learn.
Neural Netw., pp. 363-378, 1998.
[44] H. Ritter, A. Botev, and D. Barber, " Online structured Laplace approximations for
overcoming catastrophic forgetting, " in Proc. 32nd Int. Conf. Neural Inf. Process. Syst.,
(NIPS'18), 2018, pp. 3742-3752.
[45] S. Pouyanfar et al., " A survey on deep learning: Algorithms, techniques, and applications, "
ACM Comput. Surv., vol. 51, no. 5, Sep. 2018, doi: 10.1145/3234150.
[46] W. L. Buntine, " Operations for learning with graphical models, " J. Artif. Intell. Res.,
vol. 2, pp. 159-225, Dec. 1994, doi: 10.1613/jair.62.
[47] Y. Wen, P. Vicol, J. Ba, D. Tran, and R. Grosse, " Flipout: Efficient pseudo-independent
weight perturbations on mini-batches, " in Proc. Int. Conf. Learn. Representations, 2018.
[48] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, " Understanding deep
learning requires rethinking generalization, " in Proc. 5th Int. Conf. Learn. Representations,
ICLR, 2017.
MAY 2022 | IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE 47
http://www.arxiv.org/abs/2104.14421 http://www.arxiv.org/abs/2104.14421 http://www.arxiv.org/abs/2002.10118 https://www.github.com/french-paragon/BayesianNeuralNetwork-Tutorial-Metarepos https://www.github.com/french-paragon/BayesianNeuralNetwork-Tutorial-Metarepos https://www.arc.gov.au/ https://www.afrl.dodlive.mil/tag/darpa/ https://www.afrl.dodlive.mil/tag/darpa/ http://www.arxiv.org/abs/1904.11643 http://www.arxiv.org/abs/2002.08791 http://www.deeplearningbook.org
IEEE Computational Intelligence Magazine - May 2022

Table of Contents for the Digital Edition of IEEE Computational Intelligence Magazine - May 2022
