0
$\begingroup$

I'm currently using Gradient Boosting Regressor as my model to predict production risk based off a set number of features as a side-project. One of these features, company_name, has a seemingly endless amount of categorial variables (1000+ unique values).

The idea is that company_name could be useful in predicting which companies are more at-risk of not following current production standards set out by a Standards Organization. Encoding this feature through One-Hot encoding creates a LOT of new categories in the model.

I run into issues when I'm trying to run prediction through the model. Using the same dataset I used to train it (splitting it by train/test) I am successful. The issue arises when I have an upcoming schedule of production items (a new file!) and one-hot encode. The dataframe I try to predict with is simply a completely different shape of the trained model. I simply don't know proper procedure... An added piece to this problem is that sometimes I may come across a new categorical variable in the upcoming schedule that hasn't been trained on yet.

My options could be:

  1. Compare Column names and simply add all columns that don't exist in the data frame I try to predict with as 0. It's difficult as this could lead to poor performance as the new file would only contain ~100 upcoming schedules. Another issue is that the upcoming schedule may have a new company name that wasn't in company_name when we trained the model-- leading to me having to do some sort of dynamic model training every single time I want to predict with the upcoming schedule (?)
  2. Encode the model differently. I'm not sure which one is best as realistically the features I am using are all categorial variables.
  3. Don't use this feature. Since the upcoming schedules and original dataset are so different, there's only two to three features that I could realistically use. Taking one of these away would appear to greatly hamper prediction performance (I believe).

If there's anything I'm not considering/any suggestions-- that would be greatly appreciated! This can be a new model I haven't considered or a comment on anything above.

Edit for clarification: The issue particularly lies in that encoding it through via one-hot produces ValueError: X has 64 features, but GradientBoostingRegressor is expecting 4722 features as input. whenever I try and predict on the model.

This is because I'm using one-hot encoding on the new file, the upcoming production schedule, which has a very small subset of the company_name variable (or possibly new ones that weren't even seen from when we trained the model!)

I'm particularly asking on how to handle this? I'm new to this field and couldn't derive a concrete answer by searching online.

$\endgroup$
6
  • $\begingroup$ Welcome Andrew. It appears you are asking for help on more than one topic. The title suggests the issue is around one-hot-encoding but you move onto other production issues and it's unclear if they are related. Please have a look at the guidance on how best to ask a question. $\endgroup$
    – fswings
    Commented Oct 22, 2023 at 23:35
  • $\begingroup$ If the issues is around one-hot-encoding, then please add some more context. If you have 1000+ company names and 1000 rows in total, then i can understand your concern. If it's 1000+ companies but 1M rows then I don't see a problem. It would help if you stated why you think it's an issue. What's going wrong? $\endgroup$
    – fswings
    Commented Oct 22, 2023 at 23:38
  • $\begingroup$ Hey @fswings, thanks for reaching out. I did add some more context to it. It kind of had to do with one-hot encoding and I do have around ~50,000 rows in total. I believe I stated more definitively what was going wrong but please feel free to ping again for more specifications. I gave a couple solution to my problems in the options tab, and my thoughts on why I didn't implement them. This post is more of a higher-level design question as I debate my next steps. $\endgroup$ Commented Oct 23, 2023 at 0:18
  • $\begingroup$ Seems like you could save the company_names you encounter during training, and pass it as the categories argument to OneHotEncoder. That's how you'd do this in the sklearn library, but all of them have some way of doing this. $\endgroup$
    – Nick ODell
    Commented Oct 23, 2023 at 19:39
  • $\begingroup$ More broadly: why would you expect the company name to correlate with your variable of interest? Maybe the length of time the company has existed would be an interesting feature. (e.g. new companies tend to have more mistakes.) Maybe the proportion of times the company has previously failed to follow standards would be interesting feature. (e.g. the mistakes come from bad company culture.) But if you're one-hot encoding company name, I don't know what that could tell you about a company that has never appeared in your dataset. $\endgroup$
    – Nick ODell
    Commented Oct 23, 2023 at 19:44

1 Answer 1

1
$\begingroup$

Edit for clarification: The issue particularly lies in that encoding it through via one-hot produces ValueError: X has 64 features, but GradientBoostingRegressor is expecting 4722 features as input. whenever I try and predict on the model.

This suggests the number of input features (columns) for new files does not align with what your model is expecting. If the variable column length is to do exclusively with company_name then at least two treatments are:

  1. Add the missing columns in with a pre-processing step equal to 0.
  2. For new company_names either delete them as the model hasn't seen them or modify the model with an extra feature called unknown_company. Any new files that have new company_name, instead of adding new columns with one-hot encoding, just flag the unknown_company with a 1 instead.

This is because I'm using one-hot encoding on the new file, the upcoming production schedule, which has a very small subset of the company_name variable (or possibly new ones that weren't even seen from when we trained the model!)

I'm particularly asking on how to handle this? I'm new to this field and couldn't derive a concrete answer by searching online.

Instead of encoding all companies, consider only encoding the top X (i.e. 100) and convert the rest under a single feature called other_company. You will have to play with the numbers. This way, the model will be trained where the company is unknown.

You will have to make adjustments to your training, validation and test set to ensure that the latter 2 contain companies not seen in training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.