I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it before would leak test information into the training set, but I could very well be missing something. I'm also concerned that the split of the data will impact the analysis of correlation between features, though maybe in practice the likelihood of this is low. Anyone have any insight? Thanks!
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
By principle in supervised ML any decision which affects the model should be made using only the training set, in order to avoid data leakage. Following this principle requires the training/test split to always be done first.
Quite often the decision seems minor and has little chance to cause data leakage, like in your case. It's tempting to make our life easier and use the whole data for some pre-processing like this. However it's a good idea to follow this principle as strictly as possible not because the effect of data leakage would be huge, but because it would be impossible to detect it if it happens: when it happens, data leakage means that the performance on the test set is biased (overestimated), and there's simply no way to know by how much (except by collecting a new test set, of course).
Your concern that the split of the data could impact the analysis of correlation is a good example of the problem: suppose there would be a difference indeed. This means that there is data leakage, since the features are different depending directly on the test data. In this case the evaluation on the test set is likely to be biased.
Another way to explain the same idea is this: if something happens or doesn't happen in the training set by chance, then it's important to see if the model is able to focus on the real patterns and ignore the noise. The only way to obtain a reliable answer to this question is if nothing in the model depends on the test set.
-
$\begingroup$ Thank you this is very helpful and gives me some more insight on how to think about this question $\endgroup$ Jan 20, 2021 at 19:54
-
$\begingroup$ What if how you split train/test affects the results of the preprocessing step? That is, say if you split train/test two times using two random seeds and get d/t correlation between the features? Also according to this answer stats.stackexchange.com/a/239390/223656 for a similar question, you do train/test splits right before you are about to build your model (step. 4 in the question) not before it. What do you think? $\endgroup$– xabushNov 26, 2022 at 10:06
-
$\begingroup$ @xabush this is what I said in my amswer: if the train/test split affects the results, it means that there is some overfitting. It's better to know it by evaluating on a proper test set without any data leakage rather than to ignore it. I disagree strongly with this other answer, see also here, here... $\endgroup$– ErwanNov 26, 2022 at 12:42
-