1
$\begingroup$

I have a very small dataset ( 150 records) with 20 features, trying to predict a binary outcome. Due to the small size, i chose to do 10 CV instead of train/test as the train/test split.

I was wondering if i'm doing a GridsearchCV on 10-fold, getting the best parameters, and then using those parameters evaluating the performance on 10-fold - is that "legal" or overfitting? am i suppose to run the best parameters on the entire data ? or can i use 10-fold again?

also, will LOOCV suppose to give better generalization on the performance part? (not on the gridsearch)?

New contributor
XPeriment is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
1
$\begingroup$

I was wondering if i'm doing a GridsearchCV on 10-fold, getting the best parameters, and then using those parameters evaluating the performance on 10-fold - is that "legal" or overfitting? am i suppose to run the best parameters on the entire data ? or can i use 10-fold again?

I'm pretty sure you won't go to prison for it ;) But it would be incorrect due to data leakage: the data that you would use for the final evaluation would be the same which was used for learning the best parameters. So if the best parameters found during tuning (which is a kind of supervised training) happen by chance, the final performance will be artificially high.

The proper way to do that is either:

  • To keep a fresh test set aside, i.e. run the grid search with CV on the training set, then evaluate the final model on the fresh test set.
  • To use nested CV, i.e. a double CV loop: for each outer CV fold, run the grid-search inner-CV on the training set, then evaluate on the outer CV test set. Needless to say, it's a bit more complex.

also, will LOOCV suppose to give better generalization on the performance part? (not on the gridsearch)?

Leave one out CV is exactly the same as $k$-fold CV but with $k$ equal to the number of instances, so in your case it's like 150-fold CV. Advantage: more training data every time, so potentially better model; disadvantage: computational cost higher since the training/testing is repeated 150 times.

| improve this answer | |
$\endgroup$

Your Answer

XPeriment is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.