Is there a common strategy to measure if a difference-significance of two areas under two ROC curves

Question

I conduct sound detection experiments with mice. I have a stimulus sound and a "noise" sound that shoukd be ignored.

I want to measure how well the mouse ignors the noise (with respect to, say, ignoring 100% of the noise stimuli).

I have sessions with, say 200 trials of stimulus sound that sould be detected and 40 trials of noise sounds that should be ignored.

I created the confusion matrix and ROC curve (I use Matlab). Now I want to know how well my mouse is performing compared with "ignoring all noise stims".

Is there a way, or a common used formula, to get a evaluation or a confidence interval of "how good is my ROC AUC comparing to AUC(ROC) = 1"?

Thanks!

Answer 1

The AUC is a summary statistic of your dataset. As such, if you did the same experiment again you would get a slightly different value - and if repeated several times, you'd get a distribution of values. You probably want to show that your distribution rarely or never includes 0.5 (indicating random classification). When you have an empirical distribution like this, or a histogram of repeated experiments, you can just count how many of your estimates are $\approx 0.5$. If none, you're golden.

Since you don't want to do your experiment multiple times, you could 'simulate' that scenario by using samples of your data, but otherwise performing the same process. This is called 'bootstrapping'. You can use this method to construct confidence intervals around your AUC metric, or equivalently, do a statistical test to see if the metric you get is different from 0.5.

Some packages have bootstrapped confidence intervals built in to the AUC calculation, so you could just use that.