Predicting with categorical data

Question

I have a dataset which contains various columns: numerical and categorical.

Dataset here:

I was able to process the categorical data using .astype('category') and cat.codes features in Pandas dataframe as explained here in Approach #2.

def process_categorical(self, dataset):
        """

        :param dataset:
        :return:
        """
        # Label Encoding.
        for categorical_feature in LABEL_ENCODED_FEATURES:
            categorical_feature = _to_string(categorical_feature)
            dataset[categorical_feature] = dataset[categorical_feature].astype('category')
            dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes
        # Drop previous values.
        dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)
        # Rename back categories. {'to_user_cat': 'to_user' ... }
        dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)
        return dataset

I'm able to train my model and test data. When I generate my training dataset where categorical features were encoded. (Using .cat.codes), this numerical data is different than my predictions. Example:

dataset['user_agent_numerical'].value_counts()

10    1002
11     850
1      288
8       18
7       17
2       16
6       14
3        5
0        4
9        2
5        2
4        2

Which corresponds to: dataset.groupby(['user_agent']).size()

user_agent
24gt2wreg24h                4
Avaya one-X Deskphone     288
Deskphone                  16
FreePBX 1.8                 5
HiDude UA v3.81             2
SimpleSIP V4.3              2
Twilio Gateway             14
caller                     17
eyeBeam release 3006o      18
friendly-scanner            2
pplsip                   1002
sipcli/v1.8               850

When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)

ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,contact_user,callid,content_type,user_agent,source_ip,source_port,destination_port,contact_ip,contact_port
sip:[email protected],789011972592277524,13.57.9.131,78901113579131,13.57.9.131,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,163.172.120.42,60993,5060,212.129.10.158,60993

I need to do pre-processing, My prediction data contains pplsip as user-agent hence it ends up with 0, instead of 10.

How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value.

User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported here.

Complete code here.

Answer 1

Yep this is a common problem. What I would do is use SKLearns label encoder. With a similar API to SKLearn models, it can be fit to your category - meaning that any new data passed through the encoder object is encoded in the same fashion. For example

# Import the encoder
from sklearn.preprocessing import LabelEncoder

# Fit it to your training set + transform
encoder = LabelEncoder()
train[categorical_feature] = encoder.fit_transform(train[categorical_feature])

# Use it on the test set to ensure the same transformation
test[categorical_feature] = encoder.transform(test[categorical_feature])

Huge caveat # 1

This assumes every possible category is present in your training set. If you think this isn't the case, fit the encoder to a dataset that does have them all.

Huge caveat # 2

Depending on the model you're using, categoricals still aren't 'processed' when they've been label-encoded , you probably still need to one hot encode (SKLearn has a package for that too).