I'm currently using Gradient Boosting Regressor as my model to predict production risk based off a set number of features as a side-project. One of these features,
company_name, has a seemingly endless amount of categorial variables (1000+ unique values).
The idea is that
company_name could be useful in predicting which companies are more at-risk of not following current production standards set out by a Standards Organization. Encoding this feature through One-Hot encoding creates a LOT of new categories in the model.
I run into issues when I'm trying to run prediction through the model. Using the same dataset I used to train it (splitting it by train/test) I am successful. The issue arises when I have an upcoming schedule of production items (a new file!) and one-hot encode. The dataframe I try to predict with is simply a completely different shape of the trained model. I simply don't know proper procedure... An added piece to this problem is that sometimes I may come across a new categorical variable in the upcoming schedule that hasn't been trained on yet.
My options could be:
- Compare Column names and simply add all columns that don't exist in the data frame I try to predict with as 0. It's difficult as this could lead to poor performance as the new file would only contain ~100 upcoming schedules. Another issue is that the upcoming schedule may have a new company name that wasn't in
company_namewhen we trained the model-- leading to me having to do some sort of dynamic model training every single time I want to predict with the upcoming schedule (?)
- Encode the model differently. I'm not sure which one is best as realistically the features I am using are all categorial variables.
- Don't use this feature. Since the upcoming schedules and original dataset are so different, there's only two to three features that I could realistically use. Taking one of these away would appear to greatly hamper prediction performance (I believe).
If there's anything I'm not considering/any suggestions-- that would be greatly appreciated! This can be a new model I haven't considered or a comment on anything above.
Edit for clarification:
The issue particularly lies in that encoding it through via one-hot produces
ValueError: X has 64 features, but GradientBoostingRegressor is expecting 4722 features as input. whenever I try and predict on the model.
This is because I'm using one-hot encoding on the new file, the upcoming production schedule, which has a very small subset of the
company_name variable (or possibly new ones that weren't even seen from when we trained the model!)
I'm particularly asking on how to handle this? I'm new to this field and couldn't derive a concrete answer by searching online.