What is the right way of training Regression model having various categories involved?

Question

I am working on one regression problem statement and it involves multiple categories into it. I am not sure how to proceed with it, hence looking for your guidance/suggestions over it.

Suppose there are 'M' records and 'N' columns in the data and Target is a regression (numeric) output which is to be predicted by the model.

But the challenge over here is that, out of 'N' columns there is a column called as 'category' and it highly impacts the Target. This means that if 'M-1' columns are exactly the same for two records, but the 'category' variable is different, then the 'Target' may be very different.

With this being said, one naive approach is that I train a separate Linear Regression model for each 'category' available. But there are around 5000 different categories available in the column and hence creation of separate model is not possible.

All this forces me to create a single model, but how should I handle/use this 'category' column so that model understands it well and predict the target value accordingly.

My Approach:

• Since this 'category' column has 5k different categories, I can't go for label encoding.
• If I go for one hot encoding by pd.get_dummies, I will end up with lot of features in hand and with this, will my model be powerful enough at prediction side?

Is there any Machine learning model who can handle this type of data automatically? or If you can suggest how to handle this scenario, it will be very helpful

Answer 1

The question of the role of this 'category' column matters:

• If the categories are independent, i.e. instances in category X have no relation with instances in category Y, then it makes more sense to train one model for every category, because a global model might not be able to capture all the patterns for each category. In particular a linear regression model is very basic, it must represent every feature globally so if trained with all the categories it would definitely be strongly biased.
• If some/most of the patterns are shared across different categories, i.e. instances in category X have some similarities with instances in category Y, then it makes sense to train a single global model, especially if some categories have too few instances.

With some borderline problems, it's not easy to decide which of the two options is preferable. But imho it's important to at least try to answer it, because the model could completely fail if the wrong option is chosen.

For the record, training 5000 linear regression models is technically doable, but there could be an issue if some categories have too few instances.

I doubt that a linear regression model could deal properly with a categorical feature with many values, whatever the encoding. So I'd be more inclined to go with independent models anyway, or with a different algorithm than linear regression. Assuming that the second option is indeed the most reasonable for good reasons (i.e. not only because the other option takes too long), I think the category should be represented with regular one hot encoding.