# Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution?

I have a categorical variable which has thousands of values, for a dataset which has millions of records. The data is being used to create a binary classification model. I am in the early steps of feature selection, but I am trying out Random Forest, Boosted Trees, and Logistic Regression to see what works.

If I find the frequency of each category and sort that, I see that about 50 values make up the top 80%. Is it valid to condense this feature as a binary on whether the value is in that set of values or not. By 'valid', I mean is it likely that this sort of transformation retain any useful information for a model? I have a concern that sorting these categorical values which do not have any order to them creates some incorrect assumptions.

The frequecy distribution looks a little like this:

A;10%
D;5%
E;1.2%
B;1.1%
...
Z;0.004%
W;0.0037%
...


Going one step further, is it valid to profile each class in my dataset and do the same? Say Categories A-F comprise the top 80% of class 0 and Categories D-H are the top 80% of class 1. I would convert:

data_id;cat_var
1;B
2;F
3;H
4;Z


to

data_id;cat_var_top80class0;cat_var_top80class1
1;1;0
2;1;1
3;0;1
4;0;0


Adding picture to hopefully clear up this idea. In yellow are the pre-calculated distributions of cat_var (***_id in the picture) for classes 0 and 1 based on the training set. On the right shows how the transformation would be applied:

• The percentile approach is definitely something worth trying. Percentile is the value of cdf function of frequency/density distribution. I have seen people using this frequency concept for new feature generations. Since it works there, it is probably useful to your case too. From working solutions I have seen, the way is to create features as counts grouped by the categories. Frequency is proportional to the counts. So it should work the same way. For example, there are 3 categorical features, A, B, C. New features can be count_A_by_B_C, count_B_by_A_C, and count_C_by_A_C. Another approa – Diansheng Apr 4 '18 at 3:04