I am new to ML and trying to learn the nuances. I work on a binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3554.

What I currently do is

1) split the data into train(70%) and test(30%)

2) initiate a model --> logreg=LogisticRegression(random_state=41)

3) run 10 fold cv --> logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')

4) fit the model --> logreg_cv.fit(X_train_std,y_train)

5) Do prediction --> y_pred = logreg_cv.predict(X_test_std)

Now my question is, how to generate 10000 AUC scores.

I read that people usually do this get a confidence interval of their train and test performance AUC scores.

So, I would like to know how to do this?

I know that bootstrap means generate random samples with replacement from same dataset. But do we still have to split them into train and test? But this looks no different than CV. How do we generate 10000 AUC's and get a confidence interval?

Can you help?


In this question of stats exchange you can see an answer to your question of when to use bootstrap over CV.

You can see a simple tutorial of how to do Bootstrap in Python in this link

How to generate 10k AUC Scores? AUC is a performance metric and what you are going to measure is the performance of your model 10k times. For that, you have to select 10k times the number of samples that you consider and measure AUC

for i in range(0,10_000):
    sample = df.sample(df.shape[0]/10,random_state=i)
    X = df.drop(columns='target')
    y = df.target
    preds = logreg.predict(X)

  • 1
    $\begingroup$ Hi, thanks upvoted. But am trying to generate 10000 AUC scores with confidence interval. This exactly doesn't answer my updated question $\endgroup$ – The Great Jan 8 '20 at 11:53
  • $\begingroup$ I updated the answer. Let me know $\endgroup$ – Carlos Mougan Jan 8 '20 at 12:02
  • $\begingroup$ So here, my dataset in total has 4712 records. It divides them by 10, to get 471 records per sample. Right? So, all this for loop that you have should be done once I fit the model on my train data (70%)?. So this df.sample(df.shape[0]/10,random_state=i), is done on test data? I have test data records of 1414 records, so will this be split in 10k different samples?. Apologies if i am asking too much. But will it be possible to provide the steps like I have listed above. So I can clearly understand how to do this. $\endgroup$ – The Great Jan 8 '20 at 12:14
  • 1
    $\begingroup$ I understand your question but what I don't see is what are you trying to achieve with this. Simple Train Test split should tell you already how well does your model generalized. $\endgroup$ – Carlos Mougan Jan 8 '20 at 12:50
  • 1
    $\begingroup$ I'm not sure it makes sense to take 1/10 of the dataset as in your code; you might as well sample the same size, just with replacement. (See "Sample Size" in your second link.) $\endgroup$ – Ben Reiniger Jan 8 '20 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.