I am learning classification. I do have AUC-ROC curve of the data-set. How should I proceed to bootstrap the AUC? If I do the sampling of the same size as data-set then my AUC value would not be the same all the time?
Yes, sample the same size of your data set (with replacement), then find your AUC, say 10,000 times. You will then have 10,000 different AUCs which will give you some idea of how confident you are in the AUC result.
When you sample with replacement, you will only find around 63% of the entries in your new bootstrapped data set will be unique (so many of these will have 2+ identical entries in the bootstrapped data set. Every time you make a new bootstrapped sample, the 63% that are unique will change, as will how many times a record is duplicated in the bootstrapped sample. This helps extract some information on the natural variability of your data.
So to recap, if you have 50,000 records, which means 50,000 probabilities/values and 50,000 class labels:
1) samples 1:50,000 with replacement. You should find about 31,500 unique records in this new sample.
2) calculate the AUC
3) repeat steps 1 and 2, 10,000 times(pending computational times, I would shoot for at least 1000), and save those 10,000 AUCS.
4) Now you have 10,000 AUCs. If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval. The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error $(sd/sqrt(n))$ to give some idea of the variability.
Rather than just doing one AUC calculation on your full data and saying the AUC is $.77$, you may end up finding your AUC is $.75 +/- .03$, which is much more reliable to make a claim on. Now in hindsight you can see your single AUC measure was a little optimistically high.
I think sampling of the same size as data-set then your AUC value would not be the same all the time. Based on my understanding, you might misunderstand the sample with replacement. Even your sample size is the same as the data size, n, (we only talked about OOB), it means you pick 1 case each time and return it back and repeat n times to get n samples for this entire sampling. Then you could repeat enough this process for x replicas. By this process, even your sample size is the same as data size, you will not get the same items with the data set in each sampling.