1
$\begingroup$

I am assessing the accuracy of my classification model. I performed a 4-folds cross-validation and I obtained the following Overall Accuracy: OA = (0.910, 0.920, 0.880, 0.910). So, the average OA is 0.905. My dataset contains 120 samples, therefore in each fold of cross-validation I used 90 samples for training (3/4) and 30 samples for validation (1/4).

Now I want to calculate the 95% confidence interval around the mean. I am thinking of using the following formula to calculate the symmetrical interval with respect to the average (confidence interval of the binomial proportion):

interval = z * sqrt ((accuracy * (1 - accuracy)) / n)

where z is the number of standard deviations from The Gaussian Distribution and n is the number of samples. For 95% C.I. z = 1.96. But,

What I should use n value? 120, 30, 4?

Is there a better way to calculate it?

$\endgroup$
3
  • $\begingroup$ which classification model are you using ? what do you mean by OA ? $\endgroup$
    – moth
    Jan 4, 2023 at 14:21
  • $\begingroup$ It is a combination of a convolutional neural network and a Random Forest classifier. OA is the Overall Accuracy. I'm computing the following accuracy metrics: OA, Kohen's Kappa Index, and Precision and Recall by class. $\endgroup$
    – sermomon
    Jan 4, 2023 at 16:57
  • 1
    $\begingroup$ The standard CI formula will not work. A modified version is proposed in "Cross-validation Confidence Intervals for Test Error" by Bayle et al. $\endgroup$ Jan 4, 2023 at 19:12

1 Answer 1

1
+50
$\begingroup$

Here $n=120$ will be the right answer as you are calculating the distribution over the total number of trials performed accross all folds.

The confidence interval here is over a series of binary trials, which here is the per-data point classifications.

So the average trial here would be over the total number of trials in the k-fold validations, which is

$$30*4 = 120$$

from Wikipedia

Using the normal approximation, the success probability p is estimated as $$ {\displaystyle {\hat {p}}\pm z{\sqrt {\frac {{\hat > {p}}\left(1-{\hat {p}}\right)}{n}}},} $$ or the equivalent $$ > {\displaystyle {\frac {n_{S}}{n}}\pm {\frac {z}{n{\sqrt {n}}}}{\sqrt > {n_{S}n_{F}}},}{\displaystyle {\frac {n_{S}}{n}}\pm {\frac {z}{n{\sqrt > {n}}}}{\sqrt {n_{S}n_{F}}},} $$ ... measured with ${\displaystyle n}$ trials

In this case, each trial is a classification, since each prediction on validation data point is a Bernoulli trial (binary classifications) with some $n_s$ successes (correct classifications) and $n_f$ failures (incorrect classifications).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.