I conduct sound detection experiments with mice. I have a stimulus sound and a "noise" sound that shoukd be ignored.

I want to measure how well the mouse ignors the noise (with respect to, say, ignoring 100% of the noise stimuli).

I have sessions with, say 200 trials of stimulus sound that sould be detected and 40 trials of noise sounds that should be ignored.

I created the confusion matrix and ROC curve (I use Matlab). Now I want to know how well my mouse is performing compared with "ignoring all noise stims".

Is there a way, or a common used formula, to get a evaluation or a confidence interval of "how good is my ROC AUC comparing to AUC(ROC) = 1"?


enter image description here

enter image description here

  • $\begingroup$ What is your current AUC? $\endgroup$ – Bruno Lubascher Dec 3 '19 at 13:37
  • $\begingroup$ @BrunoGL, its 0.93352. You are right.. I will add to my questions the confusion matrix and ROC curve. Thank you. $\endgroup$ – user135172 Dec 3 '19 at 14:08

The AUC is a summary statistic of your dataset. As such, if you did the same experiment again you would get a slightly different value - and if repeated several times, you'd get a distribution of values. You probably want to show that your distribution rarely or never includes 0.5 (indicating random classification). When you have an empirical distribution like this, or a histogram of repeated experiments, you can just count how many of your estimates are $\approx 0.5$. If none, you're golden.

Since you don't want to do your experiment multiple times, you could 'simulate' that scenario by using samples of your data, but otherwise performing the same process. This is called 'bootstrapping'. You can use this method to construct confidence intervals around your AUC metric, or equivalently, do a statistical test to see if the metric you get is different from 0.5.

Some packages have bootstrapped confidence intervals built in to the AUC calculation, so you could just use that.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.