4
$\begingroup$

I am learning classification. I do have AUC-ROC curve of the data-set. How should I proceed to bootstrap the AUC? If I do the sampling of the same size as data-set then my AUC value would not be the same all the time?

$\endgroup$
2
  • $\begingroup$ AUC is area under the ROC curve, but it's also a property of a classifier's classification, not the data. What do you mean by bootstrap and what does it mean to take a sample of a data set that's the same size as the data set? $\endgroup$
    – Sean Owen
    Oct 5, 2016 at 13:17
  • $\begingroup$ The problem I am trying to solve is a binary classification. Samples are in the form of two columns (true/false, probability of true): (1, 0.43), (0, 0.32), (1, 0.52) etc. I did find the AUC of ROC curve for different threshold probabilities/decision boundaries. The value of AUC I found for the data-set is close to 0.77. I need to find the Confidence interval for AUC of the ROC. One way to do is to bootstrap the data with replacement. I am not sure how to do that? $\endgroup$
    – Mat_python
    Oct 5, 2016 at 16:14

3 Answers 3

5
$\begingroup$

Yes, sample the same size of your data set (with replacement), then find your AUC, say 10,000 times. You will then have 10,000 different AUCs which will give you some idea of how confident you are in the AUC result.

When you sample with replacement, you will only find around 63% of the entries in your new bootstrapped data set will be unique (so many of these will have 2+ identical entries in the bootstrapped data set. Every time you make a new bootstrapped sample, the 63% that are unique will change, as will how many times a record is duplicated in the bootstrapped sample. This helps extract some information on the natural variability of your data.

So to recap, if you have 50,000 records, which means 50,000 probabilities/values and 50,000 class labels:

1) samples 1:50,000 with replacement. You should find about 31,500 unique records in this new sample.

2) calculate the AUC

3) repeat steps 1 and 2, 10,000 times(pending computational times, I would shoot for at least 1000), and save those 10,000 AUCS.

4) Now you have 10,000 AUCs. If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval. The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error $(sd/sqrt(n))$ to give some idea of the variability.

Rather than just doing one AUC calculation on your full data and saying the AUC is $.77$, you may end up finding your AUC is $.75 +/- .03$, which is much more reliable to make a claim on. Now in hindsight you can see your single AUC measure was a little optimistically high.

$\endgroup$
1
  • 1
    $\begingroup$ Minor correction to this: for a 95% confidence interval, you should be taking the 2.5th and 97.5th percentile. Going from the 5th to 95th percentile is a 90% confidence interval. $\endgroup$
    – Aku
    Oct 1, 2021 at 18:22
1
$\begingroup$

I think sampling of the same size as data-set then your AUC value would not be the same all the time. Based on my understanding, you might misunderstand the sample with replacement. Even your sample size is the same as the data size, n, (we only talked about OOB), it means you pick 1 case each time and return it back and repeat n times to get n samples for this entire sampling. Then you could repeat enough this process for x replicas. By this process, even your sample size is the same as data size, you will not get the same items with the data set in each sampling.

$\endgroup$
0
$\begingroup$

two questions:

1, About 50000 entries do you mean 50000 test data? and you have other data to train and fine tuning the model, is that right?

2, If the data set if more than 50000 entries, you will split the data set into train, validate and test dataset(50000 entries). If you split it randomly, the AUC result should be different every time, so how could we using bootstrap method on ONE TIME Classification result to calculate 95% CI AUC?

$\endgroup$
2
  • 1
    $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Nov 29, 2021 at 16:16
  • $\begingroup$ Your Answer has Three Questions in it and No Answers AFAICT :( $\endgroup$
    – jtlz2
    Dec 11, 2023 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.