Revisit - Encoding before vs after train test split?

Question

Background: Like a good data scientist, I planned to fit my encoder on my training set and use it to transform my test set. However, when I tried to transform my test set, the encoder threw an error. The reason was that there was a class in the test set that was not encountered in the training set during fitting (due to it being a low frequency class), so the encoder did not know how to deal with it.

One may argue that if I had performed encoding on the total dataset, then the encoder would have knowledge of the class in my test set, which was not in my training set. This would be data leakage. However, I am unconvinced that data leakage--even if it exists--is problematic. In fact, the classes are known beforehand (even if they are not known to the model), so why not encode them accordingly.

Prior search: The issue of whether to perform encoding before or after the train-test split has been brought up before (see post1 and post2). I was unable to leave a comment being a newcomer, so I am raising the issue again, this time to get a deeper explanation. The second post says "it's probably not as serious as other kinds of data leakage" but does not offer support.

Question: Is the data leakage that occurs for variable encoding (one-hot encoding or ordinal encoding) problematic and how so? A reference would be appreciated.

By the way, I am not arguing that data leakage is not a problem for every situation. I agree that for some preprocessing steps--like normalization--it makes total sense to perform the fit on the training set and the transform on the test set. With normalization, the data leakage is obvious and it is clear that it would be problematic. I am just not convinced that data leakage for encoding categorical variables is a problem.

Answer 1

In my opinion, the risk of data leakage is not the main reason why it's not a good idea to encode the class on the full dataset.

The only reason why one would do the encoding on the full dataset would be to avoid exactly this issue of having an unknown class in the test set. In other words, to prevent a technical error because the code will crash when it encouters the unknown class during evaluation.

But the technical error is just the symptom, the real problem is that it makes no sense to train a supervised model and expect it to predict classes that it did not see during training:

• Either the class should really be taken into account in the task, and then the training set is clearly not representative enough.
• Or it's an anomaly and it should be completely discarded. If there is no way to extend the training set, this is the only option.

Anyway if the class appears very few times in the full dataset, it's very likely that the model doesn't have enough examples to accurately deal with this class. So even if by chance it has one or two instances in the training set, the high risk overfitting means it's not worth to take the class into account.