1
$\begingroup$

I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it before would leak test information into the training set, but I could very well be missing something. I'm also concerned that the split of the data will impact the analysis of correlation between features, though maybe in practice the likelihood of this is low. Anyone have any insight? Thanks!

$\endgroup$
2
$\begingroup$

By principle in supervised ML any decision which affects the model should be made using only the training set, in order to avoid data leakage. Following this principle requires the training/test split to always be done first.

Quite often the decision seems minor and has little chance to cause data leakage, like in your case. It's tempting to make our life easier and use the whole data for some pre-processing like this. However it's a good idea to follow this principle as strictly as possible not because the effect of data leakage would be huge, but because it would be impossible to detect it if it happens: when it happens, data leakage means that the performance on the test set is biased (overestimated), and there's simply no way to know by how much (except by collecting a new test set, of course).

Your concern that the split of the data could impact the analysis of correlation is a good example of the problem: suppose there would be a difference indeed. This means that there is data leakage, since the features are different depending directly on the test data. In this case the evaluation on the test set is likely to be biased.

Another way to explain the same idea is this: if something happens or doesn't happen in the training set by chance, then it's important to see if the model is able to focus on the real patterns and ignore the noise. The only way to obtain a reliable answer to this question is if nothing in the model depends on the test set.

$\endgroup$
1
  • $\begingroup$ Thank you this is very helpful and gives me some more insight on how to think about this question $\endgroup$ – David Stein Jan 20 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.