# Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )

I have a dataset of 23k rows of an unbalanced dataset 85/15 ratio, 10 variables ( 9 of which are categorical ) , i'm using CatBoost and XGBoost for a binary classification. I applied cv (5 iteration loop) mean target encoding on the categorical variables and i got a certain accuracy. Ordinal encoding of the categorical features is giving a better accuracy than the mean encoding. How is that possible? If my understanding is correct, mean target encoding does not only numerically encode 'object'-type variables but it orders them using their impact on the target value, and the difference between the numerically-encoded new variables is also based on the categories' impact on the target Why do GBDT's perform better on a randomly encoded variable rather than a "well-encoded" one? Over-fitting ? or do GBDT's ( catboost/xgboost ) handle the ordinal encoding well enough that mean encoding is not needed? or something else?

Here's how i'm doing cross-validation mean encoding with a smoothing value of alpha = 10

Edit : Got a slightly better result by increasing the smoothing value to 30, but mean-encoding is still underperforming compared to the ordinal one.

   ## inside the loop
means1 =  X_val.groupby(column1).bank_account.agg('mean')
nrows1 = X_val.groupby(column1).size()
score1 = (np.multiply(means1,nrows1)  + globalmean*alpha) / (nrows1+alpha)
X_val.loc[:,encoded_column1] = X_val[column1]
X_val.loc[:,encoded_column1] = X_val[column1].map(score1)
## After the loop is over, i average the encodings for each category across all folds and update the value for my new encoded column
meanz1 = train_new.groupby(column1)[encoded_column1].mean()
train_new[encoded_column1] = train_new[column1].map(meanz1).copy()

• As a first guess, target-mean encoding makes the model overfit more readily? Aug 22 '19 at 18:53
• How do you combat that? more smoothing? Aug 23 '19 at 0:36
• and does that mean that ordinal encoding on categorical features is well-handled by those classifiers? does the classifier ignore the numerical relationship between the numerically-encoded variables? and considers them independant? Aug 23 '19 at 0:58
• Tree models (er, at least the most common ones) never care about the numerical values of features, only their relative ordering, since splits are made as "X<=a vs X>a". In the random ordinal encoding then, we get splits of the categorical levels into two sets, but not every bipartition is possible, and the chosen ordering will affect the result. (Some tree models can split levels truly independently, but not XGBoost and not [I think] CatBoost.) Aug 23 '19 at 2:20
• More smoothing is worth trying at least. Possibly lump small categorical levels together before their target mapping. How many levels do your categoricals have, and how are the data distributed among them? (I'll still leave this as a comment, as I'm not sure I've got the right underlying problem.) Aug 23 '19 at 2:28