1
$\begingroup$

I'm wondering what the "best practise" approach is for finding confidence intervals when evaluation the performance of a classifier on the test set.

As far as I can see, there are two different ways of evaluating the accuracy of a metric like, say, accuracy:

  1. Evaluate the accuracy using the formula interval = z * sqrt( (error * (1 - error)) / n), where n is sample size, error is classification error (i.e. 1-accuracy) and z is a number representing multiples of gaussian standard deviations.

  2. Train split the test set into k folds and train k classifiers, leaving a different fold out for each. Then evaluating all of these on the test set and calculating mean and variance.

Intuitively, I feel like the latter would give me an estimate on how "sensitive" the performance is to changes in the data whereas the former would give allow me to compare two different models directly.

I have to say I'm a bit confused...

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

You need to distinguish between uncertainty on the prediction and uncertainty on the class.

One example, lets say that we are tossing a coin. I am 100% confident that the probability of getting "tails" is 50%

On the other hand, there is a 90% probability that tomorrow will rain but the weatherman is not very certain of this to happen.

To get this definition I will recommend to read this paper: https://arxiv.org/abs/1910.09457

In recent years the tendency has been to use ensemble methods and extract some basic statistics to calculate a given interval

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks a lot for this insight! The difference between aleatoric and epistemic uncertainty is one that is also discussed extensively in the work of Yarin Gal. As I understand it he proposes using dropout as a Monte Carlo Technique to approximate a bayesian NN and thereby establish a measure of model uncertainty on an individual prediction. What I try to do, however, is to establish a measure of uncertainty on the accuracy obtained for two different classifiers on the same (finite) labeled test set. So it's approximation uncertainty that I'm trying to quantify. Would you agree? $\endgroup$
    – Tom
    Oct 22, 2021 at 15:55
  • $\begingroup$ Monte carlo drop out as a bayesian estimation is a cool technique. I use it a lot. Yes, you are giving a confidence interval for the prediction. arxiv.org/abs/1506.02142 $\endgroup$
    – Carlos Mougan
    Oct 22, 2021 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.