Bayesian Neural networks

https://arxiv.org/pdf/2305.07759 Bayesian neural networks (BNNs) provide a principled framework for modeling epistemic uncertainty by replacing the scalar weights with probability distributions, thereby transforming point predictions into predictive distributions.

Difficulty:

Training BNNs requires significantly more computational resources than training standard neural networks. The probabilistic parameterization of network weights effectively doubles the number of parameters that must be learned, increasing the memory requirements.Each forward pass on a data batch has to be performed on a set of random weight samples drawn from the current weight distribution, and the resulting predictions, i.e., the ensemble, have to be aggregated for the backward pass. Evaluation of multiple samples per data item linearly increases computational cost and memory demand. For largescale atmospheric models with hundreds of millions to billions of parameters, these costs become prohibitive.