I have a dataset which contains various columns: numerical and categorical.
Dataset here:
I was able to process the categorical data using .astype('category')
and cat.codes
features in Pandas dataframe as explained here in Approach #2.
def process_categorical(self, dataset):
"""
:param dataset:
:return:
"""
# Label Encoding.
for categorical_feature in LABEL_ENCODED_FEATURES:
categorical_feature = _to_string(categorical_feature)
dataset[categorical_feature] = dataset[categorical_feature].astype('category')
dataset[categorical_feature + '_cat'] = dataset[categorical_feature].cat.codes
# Drop previous values.
dataset = dataset.drop(LABEL_ENCODED_FEATURES, axis=1)
# Rename back categories. {'to_user_cat': 'to_user' ... }
dataset.rename(columns={c + '_cat': c for c in LABEL_ENCODED_FEATURES}, inplace=True)
return dataset
I'm able to train my model and test data.
When I generate my training dataset where categorical features were encoded. (Using .cat.codes
), this numerical data is different than my predictions. Example:
dataset['user_agent_numerical'].value_counts()
10 1002
11 850
1 288
8 18
7 17
2 16
6 14
3 5
0 4
9 2
5 2
4 2
Which corresponds to: dataset.groupby(['user_agent']).size()
user_agent
24gt2wreg24h 4
Avaya one-X Deskphone 288
Deskphone 16
FreePBX 1.8 5
HiDude UA v3.81 2
SimpleSIP V4.3 2
Twilio Gateway 14
caller 17
eyeBeam release 3006o 18
friendly-scanner 2
pplsip 1002
sipcli/v1.8 850
When I want to do a prediction and I pass the original record which looks like this: (Header just for reference)
ruri,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,contact_user,callid,content_type,user_agent,source_ip,source_port,destination_port,contact_ip,contact_port
sip:[email protected],789011972592277524,13.57.9.131,78901113579131,13.57.9.131,1823821775,789011972592277524,78901113579131,1365624720-309058623-1808658022,application/sdp,pplsip,163.172.120.42,60993,5060,212.129.10.158,60993
I need to do pre-processing, My prediction data contains pplsip
as user-agent
hence it ends up with 0, instead of 10.
How can I pass data to my predictions that needs to be transformed to categorical values. If I tried to convert it to categorical I end up with a different class value.
User-agent is just an example, but could be IP address or called number which corresponds to very large finite set hence a dictionary does not scale. I think this is a common problem, but not sure how can I solve it. I tried using dummies Approach#3 and I end up generating additional columns which doesn't match my prediction dataset. Issue reported here.
Complete code here.
pandas.get_dummies()
and pass the categorical column name. $\endgroup$get_dummies()
on your train and test data seperately?get_dummies()
creates a dummy variable for each category in your categorical variable. If you have different categories in your train and test data then you will get a different number of columns. You need to either useget_dummies()
on the whole dataset or store all the possibilities and write additional columns for the datasets that don't contain all the possibilities, after usingget_dummies()
. $\endgroup$