how to handle different categorical values in train and test dataset?

Question

I have a dataset in which if i do

train_df["era"].value_counts()

then it will show 120 different type of categorical values and then if i do

test_df["era"].value_counts()

It do have 310 totally different categorical values like

train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

So if i will convert it to onehot encoding then it will give me 430 new features and datasize will explode. So my question is
if

train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

1. Then can we remove that feature ? Because how will model relate i can think for continuous variable but this is categorical and if it do affect performance then how ?

2. how to handle this number of different values in categorical data ?

Answer 1

Whether to remove the dimension or not would wholly depend on the kind of data you're working with and how important is the feature for your modelling task.

What you need to ask is are the labels predefined? And by that I mean is there any particular upper bound to the kind of labels you're receiving? If yes, you could simply use label encoding from sklearn. After which I'd recommending doing some feature analysis and getting the feature importance score for that label and then decide if you want to discard it or not.

Regarding your second question, you need to encode the labels before you train your model.