I am working on one regression problem statement and it involves multiple categories into it. I am not sure how to proceed with it, hence looking for your guidance/suggestions over it.
Suppose there are 'M' records and 'N' columns in the data and Target is a regression (numeric) output which is to be predicted by the model.
But the challenge over here is that, out of 'N' columns there is a column called as 'category' and it highly impacts the Target. This means that if 'M-1' columns are exactly the same for two records, but the 'category' variable is different, then the 'Target' may be very different.
With this being said, one naive approach is that I train a separate Linear Regression model for each 'category' available. But there are around 5000 different categories available in the column and hence creation of separate model is not possible.
All this forces me to create a single model, but how should I handle/use this 'category' column so that model understands it well and predict the target value accordingly.
- Since this 'category' column has 5k different categories, I can't go for label encoding.
- If I go for one hot encoding by pd.get_dummies, I will end up with lot of features in hand and with this, will my model be powerful enough at prediction side?
Is there any Machine learning model who can handle this type of data automatically? or If you can suggest how to handle this scenario, it will be very helpful