0
$\begingroup$

Hi My data is distributed like follow:
enter image description here

And I only have categorical variables on many many levels. As I need to make a regression task I thought about doing leave one out encoding on my categories. However, while my model has a very good performance on my validation (0.98 R2) on my test is kind of terrible (0.5 R2).
My colleague did the encoding before the split, reaching an R2 of 0.97 on test.
His reason is that this to cover unseen new categories and to cover all the possible targets during the encoding. Isn't this a bad practice? Or for this highly skewed data is a possible behavior to adopt?
Thanks

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

R2 and encoding before split are both can be misleading. At first, target encoding, I wouldn't recommend to do it before the split - this is basically target leak. There're at least two major downsides of doing target encoding before split:

1) Overfitting - putting knowledge about your target into features during training always tend to overfit.
2) Impossible to do the same in production. Handling unseen categories is the thing which should be supported in production as well.

Running well-done N-fold cross-validation can help you check how are you handling unseen categories "in average".

There's a good example of target encoding I always share if anyone asks me about this technique. Also for highly skewed data, I can suggest using log(y) when you encoding your feature, that may help as well.

Also, I would recommend reading about R2 (from my experience it's very bad metrics for non-linear problems):
- Why R Squared is Useless?
- another post about R2

I would rather use MAPE/MAE or some metric which fits your task. Looking at your distribution, I guess it shouldn't be sensible to outliers.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ In fact I am using the MAE, I was having a discussion with my colleague in using the R2 or the MAE or MSE. Do you also think to make different models for outliers and for non outliers values? thanks for your support. $\endgroup$
    – 3nomis
    Nov 8, 2019 at 8:21
  • 1
    $\begingroup$ Glad to help. The answer depends on how you're gonna use outputs of your model, if it's important to keep the expected value of your predictions the same as in the train dataset then it's important to keep outliers and handle them this way (because they influence expected value a lot obviously). If outliers for you are mainly noise you definitely should exclude them. It depends. Sometimes you need to find these outliers to show them special offers for example. Sometimes removing outliers from the training can improve the quality of your model as well. $\endgroup$
    – i1bgv
    Nov 8, 2019 at 14:30
  • 1
    $\begingroup$ Making two different models makes sense, I used to do that once - but you should be able to interpret these two output values, more outputs - harder interpretation (most of the time) $\endgroup$
    – i1bgv
    Nov 8, 2019 at 14:31
  • 1
    $\begingroup$ As half of my output is 1(20k out of 33k) I decide to make a model to distinguish 1 or not and a regression model only on > 1 duration data, would that be a good solution? $\endgroup$
    – 3nomis
    Nov 8, 2019 at 17:12
  • $\begingroup$ yes, it makes sense for me. also, check whether all durations are realistic or not, maybe some of them are noise. $\endgroup$
    – i1bgv
    Nov 9, 2019 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.