How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

Question

NOTE: If someone else it's wondering about this topic, I understand you're getting deeper in the Data Analysis world, so I did this question before to learn that:

You encode categorical values as INTEGERES only if you're dealing with Ordinal Classes, i.e. College degree, Customer Satisfaction Surveys as an example. Otherwise if you're dealing with Nominal Classes like, gender, colors or names, you MUST convert them with other methods since they do not specific any numerical order, most known are One-hot Encoding or Dummy variables. I encorage you to read more about them and hope this has been useful.

Check the link below to see a nice explanation: https://www.youtube.com/watch?v=9yl6-HEY7_s

This may be a simple question but I think it can be useful for beginners.

I need to run a prediction model on a test dataset, so to convert the categorical variables into categorical codes that can be handled by the random forests model I use these lines with all of them:

Train:

data_['Col1_CAT'] = data_['Col1'].astype('category')
data_['Col1_CAT'] = data_['Col1_CAT'].cat.codes

So, before running the model I have to apply the same procedure to both, the Train and Test data.

And since both datasets have the same categorical variables/columns, I think it will be useful to apply the same categorical codes to each column respectively.

However, although I'm handling the same variables on each dataset I get different codes everytime I use these two lines.

So, my question is, how can I do to get the same codes everytime I convert the same categoricals on each dataset?

Thanks for your insights and feedback.

Answer 1

First, note that Random Forests can handle categorical variables (moreover, if you have too much categories, reducing this number is a good practice). If you want to apply a filter to your data, I'd suggest you using sklearn transformers (like OneHot Encoder, Label Encoding, ... pick the one you need according to what you want to do).

In this case, you have to fit the encoder in your train dataset, and then apply it in your test. If you want to apply this in a real case, you have to save your trained encoders alongside your trained model, so you can apply the encoder directly to the new data before predicting on it, so it has the same pattern.

Here is an example with Label Encoder

from sklearn import preprocessing
train, test = ... # SEPARATE YOUR DATA AS YOU WANT
le = preprocessing.LabelEncoder()
trained_le = le.fit(train)
train = trained_le.transform(train)
test = trained_le.transform(test)

Answer 2

What @BeamsAdept answered is a better way while using sklearn.

Here is what I did only using pandas, may be useful for other use cases.

training_data[1] = pd.Categorical(training_data[1])
categories = training_data[1].cat.categories
testing_data[1] = pd.Categorical(testing_data[1], categories)

training_data[1] = training_data[1].cat.codes
testing_data[1] = testing_data[1].cat.codes