0
$\begingroup$

In the book Applied Predictive Modeling by Max Kuhn and Kjell Johnson, there is an exercise concerning the calculation of a confidence interval for model accuracy. It reads as follows.

One method for understanding the uncertainty of a test set is to use a
confidence interval. To obtain a confidence interval for the overall accuracy,
the based R function binom.test can be used. It requires the user
to input the number of samples and the number correctly classified to
calculate the interval. For example, suppose a test set sample of 20 oil
samples was set aside and 76 were used for model training. For this test
set size and a model that is about 80% accurate (16 out of 20 correct),
the confidence interval would be computed using
> binom.test(16, 20)
Exact binomial test
data: 16 and 20
number of successes = 16, number of trials = 20, p-value = 0.01182
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
0.563386 0.942666
sample estimates:
probability of success
0.8
In this case, the width of the 95% confidence interval is 37.9%. Try
different samples sizes and accuracy rates to understand the trade-off
between the uncertainty in the results, the model performance, and the
test set size.

The dataset used here contains samples belonging to 7 classes. Since this is a multiclassification problem, not a binary one, shouldn't the probability of success in the null hypothesis be equal to 1/7? Is there a reason why the authors have chosen 1/2 as the probability of success?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The author uses binom.test - The binomial test is used when an experiment has two possible outcomes and you have an idea about what the probability of success is and similar to other statistical test, you measure if the success in observed set is significantly different from what was expected.

From now it is getting more complicated : the author is trying to calculate To obtain a confidence interval for **the overall accuracy**. Therefore, the test is not the accuracy per group (each seven oil type) but rather the whole model: whether it predicts correctly or not. So you are dealing with a binary output: success vs fail and you calculate the confidence interval of that.

So I assume if you play with the data and make various model with different data size (more/less than 76 data points), you will see using more data leads to a narrower confidence interval. How confidence interval is important when it comes to predictive analysis, is another question which I encourage you to ask and follow the sister community https://stats.stackexchange.com/

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I still quite don't get why would they choose 0.5 as the probability of success. Say in a binary classification, with a test set that contains 70 and 30 observations from each class. The baseline accuracy would be 70%, so if I were to use a binomial test here, I would use 0.7 as the probability of success in the null hypothesis. Now in a multi-class classification with 7 classes, assuming equal number of examples in each class, the baseline accuracy should be 1/7%, right? Is this not how this translates to a multi-class classification? On what basis is the chosen probability of success 50%? $\endgroup$
    – Mateusz
    Commented Jun 4, 2021 at 10:14
  • $\begingroup$ @Mateusz you have two points - 1) you are right - if the data is imbalanced (70/30) the baseline is 70% - here the data for the bino.test is the output from the old-classifier-accuracy, remember the classifiers accuracy not even the classifier itself ( and the assumption is the model is not biased so tend to throw out correct and false equally. $\endgroup$
    – user702846
    Commented Jun 4, 2021 at 10:44
  • $\begingroup$ @Mateusz 2) yes it looks like a multi-class classification case - but again, the author is just looking at the classification-error (true/false). his assumption is equal number of true/false from the classification-error not the classification itself. the bino.test is not valid if there are more than 2 labels, even the library gonna give you error. I agree it is a weird case for using of bino.test because to me has nothing to od with multi-class classification itself but its error. $\endgroup$
    – user702846
    Commented Jun 4, 2021 at 10:49
  • $\begingroup$ Ok, I think I have it all cleared up now. Thank you very much! $\endgroup$
    – Mateusz
    Commented Jun 4, 2021 at 11:21
  • $\begingroup$ what should we use to get the confidence intervals for the accuracies for each of the 7 classes? $\endgroup$
    – David
    Commented Sep 9 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.