Decision Trees and Categorical Feature Labelling

Question

I am working on a decision tree model and trying to decide how best to handle categorical features. The features in my dataset are generally high in cardinality and I have found that ordinal labeling does better than dummy encoding. I wonder if I can take this a step further and if, instead of assigning random numeric codes, I could assign them based on their correlation with the target variable.

For example, let us say one of my features is sales_rep_name and I am trying to predict whether there is a large or small sale. I could rank the sales reps by the proportion of large sales and use that ranking as ordinal labeling. That way when the decision tree makes its splits, it is keeping low selling reps on one side of the tree and high selling reps on the other side of the tree. Is there a flaw in this logic? Could this lead to overfitting?

Answer 1

It totally makes sense. You can also use count encodings. So, rare values tend to have similar counts (with values like 1 or 2), so you can classify rare values together at prediction time. Common values with large counts are unlikely to have the same exact count as other values. So, the common values get their own grouping with these way. For overfitting, I believe overfitting would be handled with this way of grouping similar sales reps. The depth of trees will be lesser and it may be better for overfitting.

Answer 2

It is not useful for generalization to encode target variable information in a feature. That is data leakage, providing the model with additional information that is not available at prediction.