Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am working on a regression problem and has a categorical feature 'CATEGORY' which can take as many as 1600 categories.
On top of that, each observation can have any number of categories in that column. For eg. CATEGORY column of row 1 has 'a' (just one category 'a') but CATEGORY column of row 2 has 'a','b','x' (three categories 'a','b','x') and so on.
What is the most optimal way to deal with such feature for regression problem other than using one-hot encoder or hash vectorizer?
In this kind of case, I would look into mean/median target encoding. Basically, you take the average/median target/response value of y (based on your training set only) per each category in your feature. This new statistic per group is now your encoding. For your validation/testing data, you simply join back the "learned" statistics from your training set as a lookup table.
It tends to work extremely well for features with high cardinality. Also, suppose you do use one-hot encoding. How sparse are some of the resulting columns? If you have multiple sparse columns (can be detected easily by looking for low variance features or by simply counting how many 0's to 1's there are) then likely, these categories can be grouped.
Finally, there are algorithms that have specifically been made to handle high cardinality situations internally; in fact, CatBoost (whose motivation in its creation was in part to create a gradient boosting framework that directly supported categorical variables) to my belief heavily popularized mean/median target encoding (what I explain above) and gives you many different options with respect to encoding categorical features.
