0
$\begingroup$

I made a ML model, trained and tested it with my data containing categorical variables.
To create dummy variables I used pd.get_dummies() before the split.
I now want to use my model on previously unseen data where, of course, I need to re create my dummies. Should I do it still with pd.get_dummies()? In this way isn't the encoding lost? Any suggestion on how to do it?
Thanks

$\endgroup$
4
$\begingroup$

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data.

One can do something along these lines:

import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoder


def get_encoder_inst(feature_col):
    """
    returns: an instance of sklearn OneHotEncoder fit against a (training) column feature;
    such instance is saved and can then be loaded to transform unseen data
    """
    assert isinstance(feature_col, pd.Series)
    feature_vec = feature_col.sort_values().values.reshape(-1, 1)
    enc = OneHotEncoder(handle_unknown='ignore')
    enc.fit(feature_vec) 
    with open(file_name, 'wb') as output_file:
            pickle.dump(enc, output_file)
    return enc 

which can then be loaded and applied as

def get_one_hot_enc(feature_col, enc):
    """
    maps an unseen column feature using one-hot-encoding previously fit against training data 
    returns: a pd.DataFrame of newly one-hot-encoded feature
    """
    assert isinstance(feature_col, pd.Series)
    assert isinstance(enc, OneHotEncoder)
    unseen_vec = feature_col.values.reshape(-1, 1)
    encoded_vec = enc.transform(unseen_vec).toarray()
    encoded_df = pd.DataFrame(encoded_vec)
    return encoded_df

where the argument enc in the latter function is an instance of OneHotEncoder that you load through pickle.load. Of course the above is just a pseudo-code example, take care that all the objects that you are using keep the initial shapes and so on.

The problem with using pd.get_dummies is that it has no memory of the previously mapped encoding: it basically turns a column into factors, whereas OneHotEncoder actually maps categorical variables to a fixed-length vector representation that is stored and kept.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.