5
$\begingroup$

I built a prediction model and predicted on new data. I now want to specify a value of my confidence in this predicted value, e.g. ranging from 0 to 1. Three methods come to my mind

  1. Built 100 models on bootstrapped data, predict 100 times on each new observation, then calculate confidence intervals. Smaller intervals mean higher confidence. High computational effort.
  2. use oob predictions of a random forest from every tree
  3. Bayesian methods can give confidence intervals through the posterior

Are there other/better ones?

| improve this question | |
$\endgroup$
  • $\begingroup$ I think you've got the idea. $\endgroup$ – Emre Jan 23 '16 at 8:57
  • 1
    $\begingroup$ @spore234 Neural network models on fixed-classes and with a softmax output layer can inherently do something similar. When you know that the data is one of n given classes, then the network will output a probability distribution for all classes, given the data when you have a softmax output layer. $\endgroup$ – Martin Thoma Jan 24 '16 at 13:05
  • 1
    $\begingroup$ No @moose, that's a misconception. A distribution over the classes says nothing about the model uncertainty; the distribution has to add up to unity, after all. What if the model is not confident about any of the classes? $\endgroup$ – Emre Jan 24 '16 at 20:48
  • 1
    $\begingroup$ That is incorrect. Softmax will give you the confidence of one class relative to another, but not their absolute confidences, so they could all be uncertain and you would not know. $\endgroup$ – Emre Jan 24 '16 at 21:16
  • 1
    $\begingroup$ @spore234 You could also use a Gaussian Process classifier $\endgroup$ – Martin Thoma Jan 24 '16 at 23:14
1
$\begingroup$

If you've got enough information on the underlying distributions of your data, classical statistical theory on regression models is an obvious alternative. These models always come with nice derivations of error estimates. See Frank Harrell's "Regression Modelling Strategies" for a comprehensive overview.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Bayesian methods (e.g. Markov Chain Monte Carlo, Variational Bayes) may indeed be a good fit, but you may explore their combination with other approaches, like bayesian deep learning (e.g. this), which can add up learning intermediate representations of the information.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.