What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)?

Question

I have a data set I'm trying to create a predictor model for. The 5 features and outcome are all categorical data. One of the features contains 5,000 unique levels. While the other 4 are all under 100 levels. The output variable in my dataset contains 400 unique levels.

How do I handle this scenario for modeling?

My approach was going to be:

1. Convert the 5,000 level feature to a sequential number list via a lookup table.
2. Convert the remaining 4 features using One Hot Encoding.
3. Convert the output variable to a sequential number list via a lookup table.
4. Use xgboost algorithm.

I don't think I can bin the 5,000 level feature because they are user id's. So to me, there is no way to bin them.

Answer 1

Personally, I think that building a predictor on such user Ids is not that efficient neither useful. Can you change your problem in a different manner?

When we do any analysis we drop those ids as they are of no use they are only the incidents in a large population and we try to group them in categories which describe their features. For example, users which have good eyesight will play basketball even if it rains.

So try to think like that.

Answer 2

I don't think I can bin the 5,000 level feature because they are user id's.

Unless there are many training examples per user, using user_id as a feature will most likely result in overfitting. This should be fairly easy to verify using a tree model that can handle categorical features of high cardinality, as you suggested.

One approach I have seen in the past is to only label the top M most frequent users and label everyone else other, so you only have M+1 possible labels for the category.

Another is to cluster users into N groups based on some useful similarity measure. You will probably have to apply some domain knowledge to come up with a good similarity metric.

If you expect the top M users to be present in a significant percentage of future examples then this might be useful, otherwise it would be better to cluster users based on other features. In the latter case you compute the user's cluster membership at prediction time and use that cluster label as an input feature.

You should compare all models to one that doesn't use user_id at all to gauge its usefulness.

Convert the 5,000 level feature to a sequential number list via a lookup table.

If I understand this correctly you want to convert this feature to a single number like an integer from 1 to 5000 to reduce memory usage. This will not affect the results you get from a tree model, but most other models will now interpret the labels as having a natural ordering and an importance relative to the size of the number representing them.

There are a few possible tricks for encoding high-cardinality categorical features as a single number (or a lower-dimensional vector).

1. Encode each user_id as the number of times it appears in the training set
2. Encode each user_id as the mean of the target value in the training set. See "Target-based encoding" here. If the target is categorical you can use a vector of frequencies for each target category, eg `[0.4, 0.1, 0.3, 0.2].

As mentioned, these and other methods will only be useful if each user tends to appear multiple times in the training set.

Also see the categorical-encoding library for more ideas.

Answer 3

I'm assuming your observations($n$) are larger than 5000, otherwise, it doesn't make any sense to use 'id' as a feature. Firstly try to find some patterns, for example, your Ids could be like this:

• C56
• A85
• C14

You can extract the alphabet from the ids and create new levels like "A" and "C". Even after this, if the number of levels are very high, then as answered above: Keep the most frequent ids as it as( select a threshold, say these levels contain 80 % of the ids), and label everything as other. You can then perform one-hot encoding.

NOTE: It's usually a practice to perform one-hot encoding when the levels are around 10-15 or less, if not reduce the number of levels.