I have a data set I'm trying to create a predictor model for. The 5 features and outcome are all categorical data. One of the features contains 5,000 unique levels. While the other 4 are all under 100 levels. The output variable in my dataset contains 400 unique levels.

How do I handle this scenario for modeling?

My approach was going to be:

  1. Convert the 5,000 level feature to a sequential number list via a lookup table.
  2. Convert the remaining 4 features using One Hot Encoding.
  3. Convert the output variable to a sequential number list via a lookup table.
  4. Use xgboost algorithm.

I don't think I can bin the 5,000 level feature because they are user id's. So to me, there is no way to bin them.


2 Answers 2


Personally, I think that building a predictor on such user Ids is not that efficient neither useful. Can you change your problem in a different manner?

When we do any analysis we drop those ids as they are of no use they are only the incidents in a large population and we try to group them in categories which describe their features. For example, users which have good eyesight will play basketball even if it rains.

So try to think like that.


I don't think I can bin the 5,000 level feature because they are user id's.

Unless there are many training examples per user, using user_id as a feature will most likely result in overfitting. This should be fairly easy to verify using a tree model that can handle categorical features of high cardinality, as you suggested.

One approach I have seen in the past is to only label the top M most frequent users and label everyone else other, so you only have M+1 possible labels for the category.

Another is to cluster users into N groups based on some useful similarity measure. You will probably have to apply some domain knowledge to come up with a good similarity metric.

If you expect the top M users to be present in a significant percentage of future examples then this might be useful, otherwise it would be better to cluster users based on other features. In the latter case you compute the user's cluster membership at prediction time and use that cluster label as an input feature.

You should compare all models to one that doesn't use user_id at all to gauge its usefulness.

Convert the 5,000 level feature to a sequential number list via a lookup table.

If I understand this correctly you want to convert this feature to a single number like an integer from 1 to 5000 to reduce memory usage. This will not affect the results you get from a tree model, but most other models will now interpret the labels as having a natural ordering and an importance relative to the size of the number representing them.

There are a few possible tricks for encoding high-cardinality categorical features as a single number (or a lower-dimensional vector).

  1. Encode each user_id as the number of times it appears in the training set
  2. Encode each user_id as the mean of the target value in the training set. See "Target-based encoding" here. If the target is categorical you can use a vector of frequencies for each target category, eg `[0.4, 0.1, 0.3, 0.2].

As mentioned, these and other methods will only be useful if each user tends to appear multiple times in the training set.

Also see the categorical-encoding library for more ideas.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.