2
$\begingroup$

I am learning classification. I do have AUC-ROC curve of the data-set. How should I proceed to bootstrap the AUC? If I do the sampling of the same size as data-set then my AUC value would not be the same all the time?

$\endgroup$
  • $\begingroup$ AUC is area under the ROC curve, but it's also a property of a classifier's classification, not the data. What do you mean by bootstrap and what does it mean to take a sample of a data set that's the same size as the data set? $\endgroup$ – Sean Owen Oct 5 '16 at 13:17
  • $\begingroup$ The problem I am trying to solve is a binary classification. Samples are in the form of two columns (true/false, probability of true): (1, 0.43), (0, 0.32), (1, 0.52) etc. I did find the AUC of ROC curve for different threshold probabilities/decision boundaries. The value of AUC I found for the data-set is close to 0.77. I need to find the Confidence interval for AUC of the ROC. One way to do is to bootstrap the data with replacement. I am not sure how to do that? $\endgroup$ – Mat_python Oct 5 '16 at 16:14
3
$\begingroup$

Yes, sample the same size of your data set (with replacement), then find your AUC, say 10,000 times. You will then have 10,000 different AUCs which will give you some idea of how confident you are in the AUC result.

When you sample with replacement, you will only find around 63% of the entries in your new bootstrapped data set will be unique (so many of these will have 2+ identical entries in the bootstrapped data set. Every time you make a new bootstrapped sample, the 63% that are unique will change, as will how many times a record is duplicated in the bootstrapped sample. This helps extract some information on the natural variability of your data.

So to recap, if you have 50,000 records, which means 50,000 probabilities/values and 50,000 class labels:

1) samples 1:50,000 with replacement. You should find about 31,500 unique records in this new sample.

2) calculate the AUC

3) repeat steps 1 and 2, 10,000 times(pending computational times, I would shoot for at least 1000), and save those 10,000 AUCS.

4) Now you have 10,000 AUCs. If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval. The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error $(sd/sqrt(n))$ to give some idea of the variability.

Rather than just doing one AUC calculation on your full data and saying the AUC is $.77$, you may end up finding your AUC is $.75 +/- .03$, which is much more reliable to make a claim on. Now in hindsight you can see your single AUC measure was a little optimistically high.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I think sampling of the same size as data-set then your AUC value would not be the same all the time. Based on my understanding, you might misunderstand the sample with replacement. Even your sample size is the same as the data size, n, (we only talked about OOB), it means you pick 1 case each time and return it back and repeat n times to get n samples for this entire sampling. Then you could repeat enough this process for x replicas. By this process, even your sample size is the same as data size, you will not get the same items with the data set in each sampling.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.