How do we pre-process data for very sparse features for a decision tree?

From this Turi documentation for decision trees

It mentioned this:

Why chose decision trees? Different kinds of models have different advantages. The decision tree model is very good at handling tabular data with numerical features, or categorical features with fewer than hundreds of categories. Unlike linear models, decision trees are able to capture non-linear interaction between the features and the target.

One important note is that tree based models are not designed to work with very sparse features. When dealing with sparse input data (e.g. categorical features with large dimension), we can either pre-process the sparse features to generate numerical statistics, or switch to a linear model, which is better suited for such scenarios.

If we have a categorical column that is very sparse (many 0's), then for that category what type of numerical statistics can we transform for each row?

  • $\begingroup$ When he says sparse features he is talking about high cardinality, right? First of all, for sparse I understand that he is talking about OneHotEncoder, and you dont necessarily must preprocess data in this way to work with categorical features in tree models. The problema with high cardinality is overfitting, as you have so much categorical labels, the split points on trees will be affected and the distribution from train/test split should be different. Talking about statistics, one way you could work calculating target mean encodings. I am lcommenting because I dont know if it answers your qu $\endgroup$ – Victor Oliveira Mar 7 at 14:34
  • $\begingroup$ @VictorOliveira I mean I'm not sure what he is saying regarding statistics. If we did generate mean encodings, is this ordinal that we can put it in as a continuous variable instead of one hot encoding? If you think creating mean encoding is what he means then please put it as an answer because I never thought about this approach! $\endgroup$ – user1157751 Mar 8 at 5:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.