# Predicting with categorical data

I have a dataset which contains various columns: numerical and categorical.

Dataset here:

I was able to process the categorical data using .astype('category') and cat.codes features in Pandas dataframe as explained here in Approach #2.

def process_categorical(self, dataset):
"""

:param dataset:
:return:
"""
# Label Encoding.
for categorical_feature in LABEL_ENCODED_FEATURES:
categorical_feature = _to_string(categorical_feature)
dataset[categorical_feature] = dataset[categorical_feature].astype('category')
dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes
# Drop previous values.
dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)
# Rename back categories. {'to_user_cat': 'to_user' ... }
dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)
return dataset


I'm able to train my model and test data. When I generate my training dataset where categorical features were encoded. (Using .cat.codes), this numerical data is different than my predictions. Example:

dataset['user_agent_numerical'].value_counts()

10    1002
11     850
1      288
8       18
7       17
2       16
6       14
3        5
0        4
9        2
5        2
4        2


Which corresponds to: dataset.groupby(['user_agent']).size()

user_agent
24gt2wreg24h                4
Avaya one-X Deskphone     288
Deskphone                  16
FreePBX 1.8                 5
HiDude UA v3.81             2
SimpleSIP V4.3              2
Twilio Gateway             14
caller                     17
eyeBeam release 3006o      18
friendly-scanner            2
pplsip                   1002
sipcli/v1.8               850


When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)

ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,contact_user,callid,content_type,user_agent,source_ip,source_port,destination_port,contact_ip,contact_port
sip:789011972592277524@13.57.9.131,789011972592277524,13.57.9.131,78901113579131,13.57.9.131,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,163.172.120.42,60993,5060,212.129.10.158,60993


I need to do pre-processing, My prediction data contains pplsip as user-agent hence it ends up with 0, instead of 10.

How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value.

User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported here.

Complete code here.

• You can use pandas.get_dummies() and pass the categorical column name. – Ankit Seth Feb 22 '18 at 11:44
• I tried get_dummies and this creates additional columns in my dataset, when I pass the test data or start doing predictions dataset size doesnt match and I get an error. In original question last paragraph I shared the link to this issue. – gogasca Feb 22 '18 at 17:57
• @spicyramen, are you using get_dummies() on your train and test data seperately? get_dummies() creates a dummy variable for each category in your categorical variable. If you have different categories in your train and test data then you will get a different number of columns. You need to either use get_dummies() on the whole dataset or store all the possibilities and write additional columns for the datasets that don't contain all the possibilities, after using get_dummies(). – Stev Mar 2 '18 at 17:14

Yep this is a common problem. What I would do is use SKLearns label encoder. With a similar API to SKLearn models, it can be fit to your category - meaning that any new data passed through the encoder object is encoded in the same fashion. For example

# Import the encoder
from sklearn.preprocessing import LabelEncoder

# Fit it to your training set + transform
encoder = LabelEncoder()
train[categorical_feature] = encoder.fit_transform(train[categorical_feature])

# Use it on the test set to ensure the same transformation
test[categorical_feature] = encoder.transform(test[categorical_feature])


### Huge caveat # 1

This assumes every possible category is present in your training set. If you think this isn't the case, fit the encoder to a dataset that does have them all.

### Huge caveat # 2

Depending on the model you're using, categoricals still aren't 'processed' when they've been label-encoded , you probably still need to one hot encode (SKLearn has a package for that too).