1
$\begingroup$

I have a very small dataset ( 150 records) with 20 features, trying to predict a binary outcome. Due to the small size, i chose to do 10 CV instead of train/test as the train/test split.

I was wondering if i'm doing a GridsearchCV on 10-fold, getting the best parameters, and then using those parameters evaluating the performance on 10-fold - is that "legal" or overfitting? am i suppose to run the best parameters on the entire data ? or can i use 10-fold again?

also, will LOOCV suppose to give better generalization on the performance part? (not on the gridsearch)?

$\endgroup$
1
$\begingroup$

I was wondering if i'm doing a GridsearchCV on 10-fold, getting the best parameters, and then using those parameters evaluating the performance on 10-fold - is that "legal" or overfitting? am i suppose to run the best parameters on the entire data ? or can i use 10-fold again?

I'm pretty sure you won't go to prison for it ;) But it would be incorrect due to data leakage: the data that you would use for the final evaluation would be the same which was used for learning the best parameters. So if the best parameters found during tuning (which is a kind of supervised training) happen by chance, the final performance will be artificially high.

The proper way to do that is either:

  • To keep a fresh test set aside, i.e. run the grid search with CV on the training set, then evaluate the final model on the fresh test set.
  • To use nested CV, i.e. a double CV loop: for each outer CV fold, run the grid-search inner-CV on the training set, then evaluate on the outer CV test set. Needless to say, it's a bit more complex.

also, will LOOCV suppose to give better generalization on the performance part? (not on the gridsearch)?

Leave one out CV is exactly the same as $k$-fold CV but with $k$ equal to the number of instances, so in your case it's like 150-fold CV. Advantage: more training data every time, so potentially better model; disadvantage: computational cost higher since the training/testing is repeated 150 times.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.