1
$\begingroup$

This is more a conceptual question than related to the implementation on decision trees.

I've a feature vector say V1,V2,V3,target_variable

If the vector is a,b,c,true then, using normal decision trees, we can classify the data set.

But if the variable V1 is a set say {x,y,z},b,c,true how can I implement it?

I think of methods by using x,y,z as dimensions rather than categories but the problem is the number of categories are very huge in the order if millions. This solution won't scale.

Are there any efficient ways to deal with this problem?

Improve this question
$\endgroup$
3
  • $\begingroup$ How many rows do you have available? $\endgroup$
    – Jan van der Vegt
    Commented Jul 19, 2016 at 12:19
  • $\begingroup$ Some 110 million rows $\endgroup$
    – tourist
    Commented Jul 19, 2016 at 12:30
  • $\begingroup$ Random forest can handle continuous (x,y,z dimensions) and categorical data. If you have too many categories (I believe python sklearn can handle more than R) you could try converting to dummy variables. Also, try using the built in Gini importance to rank the variables and only use the important categories. $\endgroup$
    – Hobbes
    Commented Jul 19, 2016 at 15:32

1 Answer 1

Reset to default
1
$\begingroup$

Your intuition that a categorical feature is treated as multiple dimensions (i.e., multiple features) is correct. Millions of categories will be problematic for tree-baed algorithms that search through all features/categories for each split. A random forest would be a good model to try for your project, since it samples the features. Most software implementations let you set the sample proportion, so you could set yours really low if need be to speed up the split searches.

Alternatively you could feed your dataset into a Lasso or Elastic Net model to remove noisy variables and substantially reduce the quantity of categories under consideration. Lasso and Elastic Net have both been shown to do well on datasets where the number of features far exceeds the number of observations (e.g., dealing with genome data).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have also seen lots of sources on Lasso/Elastic Net for feature selection, but I've often gotten better results just by doing rank-sum tests and taking the variables with best (lowest) p. Or use the Gini importance. $\endgroup$
    – Hobbes
    Commented Jul 19, 2016 at 15:36
  • $\begingroup$ @Hobbes, yes, I've heard of that too. I've often seen variable importances from trees used to rank order variables and then just taking the top N variables. I also haven't seen Elastic Net or Lasso work as well in practice as it's claimed to work in theory $\endgroup$
    – Ryan Zotti
    Commented Jul 19, 2016 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.