0
$\begingroup$

I have around 10k features in my dataset and each feature is having more than 500 categories. what is the best encoding method to convert this categorical features to vector form?

"span_dir": { ""auto"": 2, "ltr": 1, "'ltr'": 1, "'rtl'": 1, ""remodels"": 0, ""ltr"": 0, ""rtl"": 0, ""sleeper"": 0, ""malefactor"": 0, ""street"": 0, ""vulcan"": 0, ""gould"": 0, ""wreathed"": 0, ""moats"": 0, ""nonperishable"": 0, ""lrt"": 0, ""churchgoer"": 0 }

"div_class": { ""ii_gt"": 8763, "gs": 8021, ""contentcontainer"": 4172, "zd-comment": 3452, ""footer-max-main-width"": 3062, ""bgcolor"": 2962, ""message_typewriter-font"": 2098, ""footer_lf_gp-fl_gp-white"": 1063, "footer_logo": 1063, ""spacer_block_block-4"": 888, ""message-htmlpart"": 852, ""footer-cta-text"": 757, ""footer-extra"": 752, ""set_outlook_hidden"": 718, ""templatecontainer"": 676, }

$\endgroup$
2
  • $\begingroup$ Can you provide more context? What kind of dataset has 10K features? Are they all categorical? What is your goal with this dataset? What approach have you considered or tried? $\endgroup$ Jan 2 at 13:58
  • $\begingroup$ Thanks for asking. I have been working on cleaning the data and have not started experimenting with any method. I was trying to come with best approach to start with. The dataset consists of html attributes of a webpage to classify it as malicious or not. Most of the features are categorical feature. I have added few examples here. $\endgroup$
    – khushi
    Jan 3 at 15:14

2 Answers 2

2
$\begingroup$

There is no single best technique for anything. You will have to try multiple techniques and see which one gives the best result. Also since your categorical variables are all high cardinal variables, I would suggest not using OneHotEncoding as it will lead to more dimensions being added.

Here is a link of some encoding techniques you can try. Try some or all of them and see which one gives the best result.

Cheers!

$\endgroup$
1
  • $\begingroup$ Nice answer. In case the link you provided becomes invalid, it might be good to add some examples in your answer and what the link is supposed to point $\endgroup$ Jan 3 at 15:32
0
$\begingroup$

As spectre said, there is no single best technique and it depends on your dataset. But for big number of categories, I have generally seen count encoding working pretty well.

I would also recommend doing a proper EDA and removing all the non significant features. Out of 10k, you probably don't need most of the features. You can check my EDA guide or any other guide for that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.