I don't think I can bin the 5,000 level feature because they are user id's.
Unless there are many training examples per user, using
user_id as a feature will most likely result in overfitting. This should be fairly easy to verify using a tree model that can handle categorical features of high cardinality, as you suggested.
One approach I have seen in the past is to only label the top
M most frequent users and label everyone else
other, so you only have
M+1 possible labels for the category.
Another is to cluster users into
N groups based on some useful similarity measure. You will probably have to apply some domain knowledge to come up with a good similarity metric.
If you expect the top
M users to be present in a significant percentage of future examples then this might be useful, otherwise it would be better to cluster users based on other features. In the latter case you compute the user's cluster membership at prediction time and use that cluster label as an input feature.
You should compare all models to one that doesn't use
user_id at all to gauge its usefulness.
Convert the 5,000 level feature to a sequential number list via a lookup table.
If I understand this correctly you want to convert this feature to a single number like an integer from
5000 to reduce memory usage. This will not affect the results you get from a tree model, but most other models will now interpret the labels as having a natural ordering and an importance relative to the size of the number representing them.
There are a few possible tricks for encoding high-cardinality categorical features as a single number (or a lower-dimensional vector).
- Encode each
user_id as the number of times it appears in the training set
- Encode each
user_id as the mean of the target value in the training set. See "Target-based encoding" here. If the target is categorical you can use a vector of frequencies for each target category, eg `[0.4, 0.1, 0.3, 0.2].
As mentioned, these and other methods will only be useful if each user tends to appear multiple times in the training set.
Also see the categorical-encoding library for more ideas.