Background: Like a good data scientist, I planned to fit my encoder on my training set and use it to transform my test set. However, when I tried to transform my test set, the encoder threw an error. The reason was that there was a class in the test set that was not encountered in the training set during fitting (due to it being a low frequency class), so the encoder did not know how to deal with it.
One may argue that if I had performed encoding on the total dataset, then the encoder would have knowledge of the class in my test set, which was not in my training set. This would be data leakage. However, I am unconvinced that data leakage--even if it exists--is problematic. In fact, the classes are known beforehand (even if they are not known to the model), so why not encode them accordingly.
Prior search: The issue of whether to perform encoding before or after the train-test split has been brought up before (see post1 and post2). I was unable to leave a comment being a newcomer, so I am raising the issue again, this time to get a deeper explanation. The second post says "it's probably not as serious as other kinds of data leakage" but does not offer support.
Question: Is the data leakage that occurs for variable encoding (one-hot encoding or ordinal encoding) problematic and how so? A reference would be appreciated.
By the way, I am not arguing that data leakage is not a problem for every situation. I agree that for some preprocessing steps--like normalization--it makes total sense to perform the fit on the training set and the transform on the test set. With normalization, the data leakage is obvious and it is clear that it would be problematic. I am just not convinced that data leakage for encoding categorical variables is a problem.