0
$\begingroup$

I have a dataset in which if i do

train_df["era"].value_counts()

then it will show 120 different type of categorical values and then if i do

test_df["era"].value_counts()

It do have 310 totally different categorical values like

train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}

So if i will convert it to onehot encoding then it will give me 430 new features and datasize will explode. So my question is
if

train_df["era"].value_counts() intersection test_df["era"].value_counts() is {}
  1. Then can we remove that feature ? Because how will model relate i can think for continuous variable but this is categorical and if it do affect performance then how ?

  2. how to handle this number of different values in categorical data ?

$\endgroup$
4
  • 2
    $\begingroup$ If there is no intersection at all, I cannot see how keeping this feature is worth it. However, if I may ask, is this feature regarding time (since is called "era"), then maybe you could represent it differently or transform to digits. $\endgroup$
    – 20roso
    Aug 24 '20 at 15:14
  • $\begingroup$ Sounds like you should do some feature engineering on this feature. I agree with @Grzegorz that it sounds like you could perhaps convert it to be numerical if it is a time feature. Could be helpful to provide more info about this feature. You may get more useful answers. $\endgroup$ Aug 24 '20 at 15:48
  • $\begingroup$ @Grzegorz Same was my thought i learn this that if it is time related then it can help. Perfect point thankyou $\endgroup$ Aug 24 '20 at 16:40
  • $\begingroup$ I would use "Boruta" to eliminate "unimportant" variables. This can turn your one-hot monster to a few hots that are quite effective. I would also suggest you look at stratification. If it turns out all levels are important then how you do your train/test split could kill your job if you don't train each case sufficiently or test each case sufficiently. $\endgroup$ Aug 24 '20 at 18:07
1
$\begingroup$

Whether to remove the dimension or not would wholly depend on the kind of data you're working with and how important is the feature for your modelling task.

What you need to ask is are the labels predefined? And by that I mean is there any particular upper bound to the kind of labels you're receiving? If yes, you could simply use label encoding from sklearn. After which I'd recommending doing some feature analysis and getting the feature importance score for that label and then decide if you want to discard it or not.

Regarding your second question, you need to encode the labels before you train your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.