# Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now want to use my model on previously unseen data where, of course, I need to re create my dummies. Should I do it still with pd.get_dummies()? In this way isn't the encoding lost? Any suggestion on how to do it?
Thanks

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data.

One can do something along these lines:

import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoder

def get_encoder_inst(feature_col):
"""
returns: an instance of sklearn OneHotEncoder fit against a (training) column feature;
such instance is saved and can then be loaded to transform unseen data
"""
assert isinstance(feature_col, pd.Series)
feature_vec = feature_col.sort_values().values.reshape(-1, 1)
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(feature_vec)
with open(file_name, 'wb') as output_file:
pickle.dump(enc, output_file)
return enc


which can then be loaded and applied as

def get_one_hot_enc(feature_col, enc):
"""
maps an unseen column feature using one-hot-encoding previously fit against training data
returns: a pd.DataFrame of newly one-hot-encoded feature
"""
assert isinstance(feature_col, pd.Series)
assert isinstance(enc, OneHotEncoder)
unseen_vec = feature_col.values.reshape(-1, 1)
encoded_vec = enc.transform(unseen_vec).toarray()
encoded_df = pd.DataFrame(encoded_vec)
return encoded_df


where the argument enc in the latter function is an instance of OneHotEncoder that you load through pickle.load. Of course the above is just a pseudo-code example, take care that all the objects that you are using keep the initial shapes and so on.

The problem with using pd.get_dummies is that it has no memory of the previously mapped encoding: it basically turns a column into factors, whereas OneHotEncoder actually maps categorical variables to a fixed-length vector representation that is stored and kept.