So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.

For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?

Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.

I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.

Much Regards

  • $\begingroup$ Can you explain more about the problem you are trying to solve? $\endgroup$ Mar 14 '19 at 11:07
  • $\begingroup$ Mainly, I'm trying to classify data according to some inputs, the inputs mainly constitute of categorical data, each categorical variable constitutes of so many distinct values. One of the independent variables is the IP address, which is essential for my classification problem. What I'm trying to do is to binary classify based on the (mostly categorical) inputs. Does that help? Let me know if you need more details. $\endgroup$ Mar 14 '19 at 11:51
  • $\begingroup$ Embedding, Domain-based-features are most promising options here. For IP, it would be subnet ID, geo-location etc. Embedding works for large number of value (Such as word embedding for 10 Million+ words) $\endgroup$ Mar 14 '19 at 11:59
  • $\begingroup$ What kind of information you are trying to extract from the IP? $\endgroup$ Mar 14 '19 at 11:59
  • $\begingroup$ @ShamitVerma My dataset already contains countries, however, the country variable might be different than the IP country (usage of VPN's/proxies for instance). I didn't know that Embeddings work for data having millions of features actually, in that case that would be a reasonable solution for my question. $\endgroup$ Mar 14 '19 at 12:04

Have you heard of CatBoostClassifier?


It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.


There are multiple options, one can try and decide which suits your data best:

  • Word Embeddings:
    • Can use pre-trained models.
    • Can train your own Word2Vec model on your domain-specific data
  • Try to group different values:
    • Rare words having very low statistical significance can be marked as other
    • Try different clustering algorithms depending upon your data

There might be other more efficient methods, will add those as well in case I find any.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.