# Calculating confidence interval for model accuracy in a multi-class classification problem

In the book Applied Predictive Modeling by Max Kuhn and Kjell Johnson, there is an exercise concerning the calculation of a confidence interval for model accuracy. It reads as follows.

One method for understanding the uncertainty of a test set is to use a
confidence interval. To obtain a confidence interval for the overall accuracy,
the based R function binom.test can be used. It requires the user
to input the number of samples and the number correctly classified to
calculate the interval. For example, suppose a test set sample of 20 oil
samples was set aside and 76 were used for model training. For this test
set size and a model that is about 80% accurate (16 out of 20 correct),
the confidence interval would be computed using
> binom.test(16, 20)
Exact binomial test
data: 16 and 20
number of successes = 16, number of trials = 20, p-value = 0.01182
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
0.563386 0.942666
sample estimates:
probability of success
0.8
In this case, the width of the 95% confidence interval is 37.9%. Try
different samples sizes and accuracy rates to understand the trade-off
between the uncertainty in the results, the model performance, and the
test set size.


The dataset used here contains samples belonging to 7 classes. Since this is a multiclassification problem, not a binary one, shouldn't the probability of success in the null hypothesis be equal to 1/7? Is there a reason why the authors have chosen 1/2 as the probability of success?

The author uses binom.test - The binomial test is used when an experiment has two possible outcomes and you have an idea about what the probability of success is and similar to other statistical test, you measure if the success in observed set is significantly different from what was expected.
From now it is getting more complicated : the author is trying to calculate To obtain a confidence interval for **the overall accuracy**. Therefore, the test is not the accuracy per group (each seven oil type) but rather the whole model: whether it predicts correctly or not. So you are dealing with a binary output: success vs fail and you calculate the confidence interval of that.