I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies()
before the split.
I now want to use my model on previously unseen data where, of course, I need to re create my dummies. Should I do it still with pd.get_dummies()
? In this way isn't the encoding lost? Any suggestion on how to do it?
Thanks
1 Answer
Yes, the encoding would be lost. You should instead use sklearn
OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data.
One can do something along these lines:
import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoder
def get_encoder_inst(feature_col):
"""
returns: an instance of sklearn OneHotEncoder fit against a (training) column feature;
such instance is saved and can then be loaded to transform unseen data
"""
assert isinstance(feature_col, pd.Series)
feature_vec = feature_col.sort_values().values.reshape(-1, 1)
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(feature_vec)
with open(file_name, 'wb') as output_file:
pickle.dump(enc, output_file)
return enc
which can then be loaded and applied as
def get_one_hot_enc(feature_col, enc):
"""
maps an unseen column feature using one-hot-encoding previously fit against training data
returns: a pd.DataFrame of newly one-hot-encoded feature
"""
assert isinstance(feature_col, pd.Series)
assert isinstance(enc, OneHotEncoder)
unseen_vec = feature_col.values.reshape(-1, 1)
encoded_vec = enc.transform(unseen_vec).toarray()
encoded_df = pd.DataFrame(encoded_vec)
return encoded_df
where the argument enc
in the latter function is an instance of OneHotEncoder
that you load through pickle.load
. Of course the above is just a pseudo-code example, take care that all the objects that you are using keep the initial shapes and so on.
The problem with using pd.get_dummies
is that it has no memory of the previously mapped encoding: it basically turns a column into factors, whereas OneHotEncoder
actually maps categorical variables to a fixed-length vector representation that is stored and kept.