I have a dataset which contains various columns: numerical and categorical.

Dataset here:

I was able to process the categorical data using .astype('category') and cat.codes features in Pandas dataframe as explained here in Approach #2.

def process_categorical(self, dataset):

        :param dataset:
        # Label Encoding.
        for categorical_feature in LABEL_ENCODED_FEATURES:
            categorical_feature = _to_string(categorical_feature)
            dataset[categorical_feature] = dataset[categorical_feature].astype('category')
            dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes
        # Drop previous values.
        dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)
        # Rename back categories. {'to_user_cat': 'to_user' ... }
        dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)
        return dataset

I'm able to train my model and test data. When I generate my training dataset where categorical features were encoded. (Using .cat.codes), this numerical data is different than my predictions. Example:


10    1002
11     850
1      288
8       18
7       17
2       16
6       14
3        5
0        4
9        2
5        2
4        2

Which corresponds to: dataset.groupby(['user_agent']).size()

24gt2wreg24h                4
Avaya one-X Deskphone     288
Deskphone                  16
FreePBX 1.8                 5
HiDude UA v3.81             2
SimpleSIP V4.3              2
Twilio Gateway             14
caller                     17
eyeBeam release 3006o      18
friendly-scanner            2
pplsip                   1002
sipcli/v1.8               850

When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)

sip:[email protected],789011972592277524,,78901113579131,,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,,60993,5060,,60993

I need to do pre-processing, My prediction data contains pplsip as user-agent hence it ends up with 0, instead of 10.

How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value.

User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported here.

Complete code here.

  • $\begingroup$ You can use pandas.get_dummies() and pass the categorical column name. $\endgroup$
    – Ankit Seth
    Feb 22, 2018 at 11:44
  • $\begingroup$ I tried get_dummies and this creates additional columns in my dataset, when I pass the test data or start doing predictions dataset size doesnt match and I get an error. In original question last paragraph I shared the link to this issue. $\endgroup$
    – gogasca
    Feb 22, 2018 at 17:57
  • 1
    $\begingroup$ @spicyramen, are you using get_dummies() on your train and test data seperately? get_dummies() creates a dummy variable for each category in your categorical variable. If you have different categories in your train and test data then you will get a different number of columns. You need to either use get_dummies() on the whole dataset or store all the possibilities and write additional columns for the datasets that don't contain all the possibilities, after using get_dummies(). $\endgroup$
    – Stev
    Mar 2, 2018 at 17:14

1 Answer 1


Yep this is a common problem. What I would do is use SKLearns label encoder. With a similar API to SKLearn models, it can be fit to your category - meaning that any new data passed through the encoder object is encoded in the same fashion. For example

# Import the encoder
from sklearn.preprocessing import LabelEncoder

# Fit it to your training set + transform
encoder = LabelEncoder()
train[categorical_feature] = encoder.fit_transform(train[categorical_feature])

# Use it on the test set to ensure the same transformation
test[categorical_feature] = encoder.transform(test[categorical_feature])

Huge caveat # 1

This assumes every possible category is present in your training set. If you think this isn't the case, fit the encoder to a dataset that does have them all.

Huge caveat # 2

Depending on the model you're using, categoricals still aren't 'processed' when they've been label-encoded , you probably still need to one hot encode (SKLearn has a package for that too).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.