1
$\begingroup$

I am doing some work with R and after obtaining the confusion matrix I have obtained the following metrics corresponding to a logistic regression:

Accuracy : 0.7763         
                 95% CI : (0.6662, 0.864)
    No Information Rate : 0.5395         
    P-Value [Acc > NIR] : 1.629e-05

And it is not clear to me how CI would be interpreted.

Maybe it would be that the Accuracy can take values between 0.666 and 0.864? What does it mean that the CI are so large?

If someone could clarify it to me I would appreciate it. Best regards.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What have you researched so far? See en.wikipedia.org/wiki/Confidence_interval $\endgroup$
    – user2974951
    Commented Apr 12, 2022 at 11:09
  • $\begingroup$ The below answers are good but missing one important element. The cut off value. Accuracy is obtained by creating a cut off value and labeling predictions for observations above that value as 1 or an event, and all observations lower than that value as 0 or a non-event. If you change the cut off value, you may change the accuracy. Hence the 95% CI for accuracy is at x cut off value. The cut off value is optimized by balancing costs of FP and FN with benefits of TP and TN. That is if your problem requires a cut off value. If not, do not even use accuracy as a metric. $\endgroup$
    – Craig
    Commented Apr 14, 2022 at 16:18

2 Answers 2

Reset to default
1
$\begingroup$

The other answer is correct and summed it up nicely: we are 95% confident the accuracy value will fall somewhere between .666 and .864. It is a probability claim for how representative your number is.

To your other question, it could mean a couple different things based on your data what the CI means. In general, the larger the CI, the bigger the range for your numbers will be. For example, we can be 95% certain the accuracy falls between .666 and .864, but you may change the CI to 99% and it might give you a value like .333 and 1.264 or something.

When you have a large range in your CI, it usually means you have high variability in your data (some of your datapoints are around .666 while others are around .864). The more data you have and the more related the data is, the lower the range will become.

It depends for what purpose you are crunching these numbers for, but generally speaking, Higher confidence intervals like 95% give you more certainty in the data while a smaller confidence interval like 75% can compile more neatly digestible graphs while sacrificing some accuracy.

Hope this helps a bit! :)

Improve this answer
$\endgroup$
0
$\begingroup$

It tells you that with 95% probability the accuracy is estimated to lie between those bounds. Thus, when repeating the experiment very often, computing the accuracies of each individual run, you would obtain values of the accuracies outside these bounds in only 5% of the cases.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ That's not quite right - the 95% CI indicates that if the experiment were repeated many times, the true unknown accuracy would lie within the 95% CI 95% of the time. A CI uses observed data to suggest where the true value might be, I don't think you can directly use it to guess where other observed values might fall. Other observed values are indeed likely to be within a 95% CI, but I'm not sure saying that likelihood is 95% is a correct interpretation. $\endgroup$
    – Nuclear Hoagie
    Commented Apr 12, 2022 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.