I have built a machine learning classifier using Sklearn and pandas as my main tools. Now, one of the input features to the model is country (to letter country code such as US). I have fit a model using the pd.get_dummies function.

Now I want to run inference on the data but a few of the countries haven't appeared in my dataset over the past months, so the pd.get_dummies function is misaligned with the fitted model.

How can this dealt with?


I think the answer to this question will solve your problem.

import pandas as pd

train = pd.DataFrame(data = [['a', 123, 'ab'], ['b', 234, 'bc']],
                     columns=['col1', 'col2', 'col3'])
test = pd.DataFrame(data = [['c', 345, 'ab'], ['b', 456, 'ab']],
                     columns=['col1', 'col2', 'col3'])

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset_preprocessed = pd.get_dummies(dataset)
train_preprocessed = dataset_preprocessed[:train_objs_num]
test_preprocessed = dataset_preprocessed[train_objs_num:]

If that doesn't help, a simple solution would be to add columns of all zeros for each country that the new dataset it missing. This will give you the correct shape of data that your classifier is expecting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.