How is uncertainty evaluated for results obtained via machine learning techniques?

Question

As machine learning (in its various forms) grows ever more ubiquitous in the sciences, it becomes important to establish logical and systematic ways to interpret machine learning results. While modern ML techniques have shown themselves to be capable of competing with or even exceeding the accuracy of more "classical" techniques, the numerical result obtained in any data analysis is only half of the story.

There are established and well formalized (mathematically) ways to evaluate uncertainties in results obtained via classical methods. How are uncertainties evaluated in a machine learning result? For example, it is (at least notionally) relatively straight forward to estimate uncertainties for fit parameters in something like a classical regression analysis. I can make some measurements, fit them to some equation, estimate some physical parameter, and e.g. estimate its uncertainty with rules following from the Gaussian error approximation. How might one determine the uncertainty in the same parameter as estimated by some machine learning algorithm?

I recognize that this likely differs with the specifics of the problem at hand and the algorithm used. Unfortunately, a simple Google search turns up mostly "hand-wavy" explanations, and I can't seem to turn up a sufficiently understandable scientific paper discussing an ML result with an in-depth discussion of uncertainty estimation.

Answer 1

Here are some of the main approaches I'm aware of. One method is to use Bayesian machine learning, which learns a probability distribution over the entire parameter space (see Joris Baan's A Comprehensive Introduction to Bayesian Deep Learning). However, these methods tend to be computationally expensive.

For classification problems, the most common approach is to use a classifier that can output class probabilities (such as the cross-entropy loss). While this probability can be interpreted as uncertainty, it usually is not well calibrated. By calibrated we mean the model uncertainty reflects the prediction results. For example we would expect that 80% of samples that are predicted with >= 80% certainty are correctly classified. So a calibration step can be added after the classification step.

For regression problems, a naïve approach is to train multiple models, either by bagging, or for deep learning models, using different weight initialisations. Then the variance of the predictions from each model can be interpreted as the uncertainty. For instance, we can use an ensemble, where we also use the mean of the predictions as the ensemble prediction. However, this gives over-confident estimations of uncertainty.

For deep learning models, there are a couple of other approaches that I am aware of.

The first is called Monte-Carlo drop-out (Gal and Ghahramani's Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning), which can be applied to any deep learning model that uses dropout. This method uses the randomness from dropout to estimate variance or uncertainty in predictions and can be applied to both regression and classification models.

The next is to change the loss function to the negative log likelihood (NLL) function. When used for regression, it provides an estimate of both the mean and variance. So models using this method have two outputs - one for the mean and the other for the variance. An early work on this is Nix and Weigend's Estimating the mean and variance of the target probability distribution, which uses separate MLPs for the mean and variance. A more recent work (Lakshminarayanan et al.'s Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles) applies this technique to any neural network, and combines it with ensembling, which further improves the uncertainty estimates.

Answer 2

I am also not aware of any papers that 'prove' that ML, in general, works. Many techniques are based on optimization, distance measures, that really do not have any probability counterparts.

I am also very suspect of some algorithms that output probabilities, unless they are based on statistical distributions. But as was implied by some of the other answers, I think that for some techniques, simulation might be the right way to go. Decision trees at least have some statistical basis in the chi square distribution. e.g and I think the development of Random Forest was an attempt to simulate some probabilities, although probabilities in ML are often internal to the algorithm and don't really generalize to what we usually think of as a statistical prediction probability