# how to handle different categorical values in train and test dataset?

I have a dataset in which if i do

train_df["era"].value_counts()


then it will show 120 different type of categorical values and then if i do

test_df["era"].value_counts()


It do have 310 totally different categorical values like

train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}


So if i will convert it to onehot encoding then it will give me 430 new features and datasize will explode. So my question is
if

train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

1. Then can we remove that feature ? Because how will model relate i can think for continuous variable but this is categorical and if it do affect performance then how ?

2. how to handle this number of different values in categorical data ?

• If there is no intersection at all, I cannot see how keeping this feature is worth it. However, if I may ask, is this feature regarding time (since is called "era"), then maybe you could represent it differently or transform to digits. – 20roso Aug 24 '20 at 15:14
• Sounds like you should do some feature engineering on this feature. I agree with @Grzegorz that it sounds like you could perhaps convert it to be numerical if it is a time feature. Could be helpful to provide more info about this feature. You may get more useful answers. – fractalnature Aug 24 '20 at 15:48
• @Grzegorz Same was my thought i learn this that if it is time related then it can help. Perfect point thankyou – Rajan Lagah Aug 24 '20 at 16:40
• I would use "Boruta" to eliminate "unimportant" variables. This can turn your one-hot monster to a few hots that are quite effective. I would also suggest you look at stratification. If it turns out all levels are important then how you do your train/test split could kill your job if you don't train each case sufficiently or test each case sufficiently. – EngrStudent Aug 24 '20 at 18:07