6
$\begingroup$

I built a prediction model and predicted on new data. I now want to specify a value of my confidence in this predicted value, e.g. ranging from 0 to 1. Three methods come to my mind

  1. Built 100 models on bootstrapped data, predict 100 times on each new observation, then calculate confidence intervals. Smaller intervals mean higher confidence. High computational effort.
  2. use oob predictions of a random forest from every tree
  3. Bayesian methods can give confidence intervals through the posterior

Are there other/better ones?

$\endgroup$
8
  • $\begingroup$ I think you've got the idea. $\endgroup$
    – Emre
    Jan 23, 2016 at 8:57
  • 1
    $\begingroup$ @spore234 Neural network models on fixed-classes and with a softmax output layer can inherently do something similar. When you know that the data is one of n given classes, then the network will output a probability distribution for all classes, given the data when you have a softmax output layer. $\endgroup$ Jan 24, 2016 at 13:05
  • 1
    $\begingroup$ No @moose, that's a misconception. A distribution over the classes says nothing about the model uncertainty; the distribution has to add up to unity, after all. What if the model is not confident about any of the classes? $\endgroup$
    – Emre
    Jan 24, 2016 at 20:48
  • 1
    $\begingroup$ That is incorrect. Softmax will give you the confidence of one class relative to another, but not their absolute confidences, so they could all be uncertain and you would not know. $\endgroup$
    – Emre
    Jan 24, 2016 at 21:16
  • 1
    $\begingroup$ @spore234 You could also use a Gaussian Process classifier $\endgroup$ Jan 24, 2016 at 23:14

2 Answers 2

1
$\begingroup$

If you've got enough information on the underlying distributions of your data, classical statistical theory on regression models is an obvious alternative. These models always come with nice derivations of error estimates. See Frank Harrell's "Regression Modelling Strategies" for a comprehensive overview.

$\endgroup$
1
$\begingroup$

Bayesian methods (e.g. Markov Chain Monte Carlo, Variational Bayes) may indeed be a good fit, but you may explore their combination with other approaches, like bayesian deep learning (e.g. this), which can add up learning intermediate representations of the information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.