So to perform my feature selection I ran cross validation over and over again, each time trying different subsets of my attributes and repeated this until I got the best cross validation score I could get. Is this alright to do or I am creating a major bias? I suspect that this could cause a bias and possibly result in data leakage because I am probably learning something about my test set by doing this, but how bad of a bias would this be? My data set is too small to create another validation set.

  • $\begingroup$ Why not use all of the features? $\endgroup$
    – Dave
    Dec 8, 2021 at 3:11
  • $\begingroup$ I tried using all of the features with a ridge regression but it severely overfit. My dataset is very small and I only have 50 data points. I have run the cross validation multiple times and it seems that choosing smaller subsets helps produce better cross validation scores. $\endgroup$ Dec 8, 2021 at 4:13

1 Answer 1


The method itself is good, it's an optimization search over the possible features subsets. This is often done with exhaustive search or genetic search.

But you have the right intuition: at the end of this process, once you have picked the best subset of features, you must evaluate on an independent test set made of unseen data. The selection of the best subset of features is a form of training, so the performance that you obtain with CV is equivalent to performance on the training set.

It's impossible to know how bad it can be without evaluating on a fresh test set. But in general if you try a very large number of subsets it's unavoidable that there's some chance in the process, meaning the performance of the best subset is very likely to be overestimated (overfitting).

  • $\begingroup$ Ok I should add I tried to do a nested cross validation and the outer fold accuracy was quite bad. I then tried to do feature selection and parameter selection using the inner folds, but they were very unstable. All of this suggest that that optimization is not possible. So at the point I just tried to pick the features that occurred most frequently from the different folds to use for my final cross validation model. I then fined tuned the features and my model parameters until I got the best cross validation score. If optimization is not possible is this the best approach? $\endgroup$ Dec 8, 2021 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.