Hi My data is distributed like follow:
And I only have categorical variables on many many levels.
As I need to make a regression task I thought about doing leave one out encoding on my categories. However, while my model has a very good performance on my validation (0.98 R2) on my test is kind of terrible (0.5 R2).
My colleague did the encoding before the split, reaching an R2 of 0.97 on test.
His reason is that this to cover unseen new categories and to cover all the possible targets during the encoding.
Isn't this a bad practice? Or for this highly skewed data is a possible behavior to adopt?
Thanks