Currently I'm doing some simple feature selection based on correlation between features and variance within one feature. I'm applying this on the whole dataset used for model building before creating the cross-validation.

My question now is if this is acceptable workflow or can significantly affect the CV stats suggesting a better model than it actually is?

Is it technically better to do the CV-split and only then select features on the training set for that split to not leak information?


If you do not use target for feature selection, then there is no leakage and you can apply this on the whole dataset. The methods that you mention: checking correlation between features and checking variance of each feature belong here.

If you do use target for feature selection, then you should use 'train' subset only. Otherwise there is a leakage. The examples here would be (i) you check correlation between each feature and target (ii) use build Random Forest model and take best features


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.