18
$\begingroup$

I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns?

I found this interesting link.

But they are converting to class/object which I don't want. I want my final output as a data frame so that I can test with different machine learning models? Or, is there any way I can use the generated matrix to train the other machine learning models other than Logistic regression or XGBoost?

Is there anyway I can implement it?

Improve this question
$\endgroup$
5
  • $\begingroup$ This is really a problem ? Assuming you are using sparse matrix representation, onehot encoding is not going to be a real problem. $\endgroup$
    – Louis T
    Nov 15, 2017 at 0:44
  • $\begingroup$ @LouisT I have millions of records more than 500m records $\endgroup$
    – vinaykva
    Nov 15, 2017 at 13:11
  • $\begingroup$ Still if its one hot, it's not going to matter if use sparse matrix $\endgroup$
    – Louis T
    Nov 15, 2017 at 13:15
  • $\begingroup$ @LouisT Complexity will increase and train time will also increase and my data will become way too sparse $\endgroup$
    – vinaykva
    Nov 15, 2017 at 14:16
  • 2
    $\begingroup$ your post is about 1 year ago, and perhaps not interested anymore, but in case still interested have you heard about Entity Embedding using Neural Nets? medium.com/@satnalikamayank12/… $\endgroup$
    – TwinPenguins
    Dec 15, 2018 at 12:45

4 Answers 4

Reset to default
14
$\begingroup$

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power.

I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian. See e.g. the paper Deep Knowledge Tracing, which says this approach is motivated by the idea of compressed sensing:

BARANIUK, R. Compressive sensing. IEEE signal processing magazine 24, 4 (2007).

Specifically, they map each vector of length N to a shorter vector of length log2(N). I have not done this myself but I think it would be worth trying.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I am wondering how to calculate distance with such encoding? $\endgroup$
    – eric2323223
    Apr 9, 2018 at 4:20
2
$\begingroup$

You can read the data and first get a list of all the unique values of your categorical variables. Then you can fit a one hot encoder object (like the sklearn.preprocessing.CategoricalEncoder) on your list of unique values.

This method can also help in a train test framework or when you are reading your data in chunks. I have created a python module that does all this on its own. You can find it in this GitHub repository - dummyPy

A short a tutorial on this - How to One Hot Encode Categorical Variables in Python?

Improve this answer
$\endgroup$
1
$\begingroup$

Refer to this link (this is also related to categorical feature having quite a few unique values):

For embedding, you can refer to this link (not written by me, but worthy to read once).

Improve this answer
$\endgroup$
0
$\begingroup$

You can do bucketing of similar values, so that values (or columns) that holds closest value (or has much similar) pattern can be replaced by one value (or column) and thus your 1600 values can come down to say 400 (or even less).

Ex. for values like (cloud like - Nimbus Clouds, drizzle, light rain, rain, heavy rain can be converted to ( light rain, rain, heavy rain).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.