I am new to ML and trying to learn the nuances. I work on a binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3554.
What I currently do is
1) split the data into train(70%) and test(30%)
2) initiate a model -->
3) run 10 fold cv -->
4) fit the model -->
5) Do prediction -->
y_pred = logreg_cv.predict(X_test_std)
Now my question is, how to generate 10000 AUC scores.
I read that people usually do this get a confidence interval of their train and test performance AUC scores.
So, I would like to know how to do this?
I know that bootstrap means generate random samples with replacement from same dataset. But do we still have to split them into train and test? But this looks no different than CV. How do we generate 10000 AUC's and get a confidence interval?
Can you help?