1
$\begingroup$

I have a classification dataset ,where I have a lot of categorical columns . I have one hot encoded ie. dummy variables in my training . How to handle this in production side of ML. There are cases in which there is drift in data in future datasets which can introduce new variables outside categories used during training the model.

What I did was after one hot encoding off all the features I saved the categorical columns and saved those as pickle file and later loaded the pickle file to match the production set features during deployment and remove the extras.

How is it done in production , the correct way?

$\endgroup$

1 Answer 1

0
$\begingroup$

This seems like reasonable for your problem. But sometimes you also face cases where some categories maybe missing and can lead to error in production. How it handles in production was little different :

  1. Pickle a pandas training dataframe used for production model with single row to ensure same columns

  2. Once you do one hot encoding on production dataframe append it to pickle dataframe so as to ensure same columns as training df. Remove the first row which cam from training. This ensures that no column used during training is missing and extra columns are ignored

$\endgroup$
1
  • $\begingroup$ Its not any different to my approach $\endgroup$
    – Shubh
    Apr 8, 2022 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.