Timeline for Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )
Current License: CC BY-SA 4.0
22 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 20, 2020 at 13:44 | comment | added | Blenz | Not much honestly. This is an old post, i remember i eventually used ordinal encoding as target encoding was performing worse. | |
| Mar 19, 2020 at 23:25 | comment | added | Lucas Morin | @Blenz: not sure this will be helpfull, but I have found target encoding to be of little or no help with advanced tree methods such as xgboost. My reasoning was that the xgboost would calculate such mean target and split variables accordingly... You would merely gain one or two split. I quite surpised it had such impact for you. How much did it impact accuracy ? | |
| Aug 28, 2019 at 13:51 | history | edited | Blenz | CC BY-SA 4.0 |
deleted 319 characters in body
|
| Aug 26, 2019 at 15:46 | history | edited | Blenz | CC BY-SA 4.0 |
added 93 characters in body
|
| Aug 26, 2019 at 14:06 | history | edited | Blenz | CC BY-SA 4.0 |
added 6 characters in body
|
| Aug 26, 2019 at 14:00 | history | edited | Blenz | CC BY-SA 4.0 |
added 6 characters in body
|
| Aug 23, 2019 at 20:06 | comment | added | silverstone | Try to use sklearn's categorical-encoder. Also you can try to increase fold count or you can use inner folds. Check this out. | |
| Aug 23, 2019 at 17:20 | history | edited | Blenz | CC BY-SA 4.0 |
deleted 1 character in body
|
| Aug 23, 2019 at 14:58 | history | edited | Blenz | CC BY-SA 4.0 |
added 143 characters in body
|
| Aug 23, 2019 at 14:29 | comment | added | Blenz | But the issue here is , overfitting or not, xgboost is able to ignore the numerical differences between the randomy numerically-encoded variables and that's something i always heard its contrary on forums. | |
| Aug 23, 2019 at 14:28 | comment | added | Blenz | Still, you're right in a way for that reasoning, because the distribution of the categories for some variables is very unbalanced 10k occurences for 1 category and 15 for a category. I caught that category unbalance could cause overfitting so i imputed the really low categories with most freq. Still same problem. | |
| Aug 23, 2019 at 14:24 | comment | added | Ben Reiniger♦ | Hrm, that's not as many levels as I would expect to be contributing to overfitting in this way... | |
| Aug 23, 2019 at 10:29 | history | edited | Blenz | CC BY-SA 4.0 |
added 141 characters in body
|
| Aug 23, 2019 at 8:29 | history | edited | Blenz | CC BY-SA 4.0 |
added 14 characters in body
|
| Aug 23, 2019 at 8:24 | comment | added | Blenz | I think you got a point with your reasoning on categorical levels. I have 3 variables that are bicategorical, the rest of my variables have from 4 to 7 categories. I also have 2 numerical variables that i'm binning for this matter. | |
| Aug 23, 2019 at 2:28 | comment | added | Ben Reiniger♦ | More smoothing is worth trying at least. Possibly lump small categorical levels together before their target mapping. How many levels do your categoricals have, and how are the data distributed among them? (I'll still leave this as a comment, as I'm not sure I've got the right underlying problem.) | |
| Aug 23, 2019 at 2:20 | comment | added | Ben Reiniger♦ | Tree models (er, at least the most common ones) never care about the numerical values of features, only their relative ordering, since splits are made as "X<=a vs X>a". In the random ordinal encoding then, we get splits of the categorical levels into two sets, but not every bipartition is possible, and the chosen ordering will affect the result. (Some tree models can split levels truly independently, but not XGBoost and not [I think] CatBoost.) | |
| Aug 23, 2019 at 0:58 | comment | added | Blenz | and does that mean that ordinal encoding on categorical features is well-handled by those classifiers? does the classifier ignore the numerical relationship between the numerically-encoded variables? and considers them independant? | |
| Aug 23, 2019 at 0:36 | comment | added | Blenz | How do you combat that? more smoothing? | |
| Aug 22, 2019 at 18:53 | comment | added | Ben Reiniger♦ | As a first guess, target-mean encoding makes the model overfit more readily? | |
| Aug 22, 2019 at 17:18 | history | edited | Blenz | CC BY-SA 4.0 |
added 699 characters in body
|
| Aug 22, 2019 at 17:10 | history | asked | Blenz | CC BY-SA 4.0 |