2
$\begingroup$

How do we pre-process data for very sparse features for a decision tree?

From this Turi documentation for decision trees

It mentioned this:

Why chose decision trees? Different kinds of models have different advantages. The decision tree model is very good at handling tabular data with numerical features, or categorical features with fewer than hundreds of categories. Unlike linear models, decision trees are able to capture non-linear interaction between the features and the target.

One important note is that tree based models are not designed to work with very sparse features. When dealing with sparse input data (e.g. categorical features with large dimension), we can either pre-process the sparse features to generate numerical statistics, or switch to a linear model, which is better suited for such scenarios.

If we have a categorical column that is very sparse (many 0's), then for that category what type of numerical statistics can we transform for each row?

Improve this question
$\endgroup$
4
  • $\begingroup$ When he says sparse features he is talking about high cardinality, right? First of all, for sparse I understand that he is talking about OneHotEncoder, and you dont necessarily must preprocess data in this way to work with categorical features in tree models. The problema with high cardinality is overfitting, as you have so much categorical labels, the split points on trees will be affected and the distribution from train/test split should be different. Talking about statistics, one way you could work calculating target mean encodings. I am lcommenting because I dont know if it answers your qu $\endgroup$
    – Victor Oliveira
    Mar 7, 2019 at 14:34
  • $\begingroup$ @VictorOliveira I mean I'm not sure what he is saying regarding statistics. If we did generate mean encodings, is this ordinal that we can put it in as a continuous variable instead of one hot encoding? If you think creating mean encoding is what he means then please put it as an answer because I never thought about this approach! $\endgroup$
    – user1157751
    Mar 8, 2019 at 5:29
  • $\begingroup$ Maybe by "numerical statistics" he means using numerical methods like SVD/PCA to do dimensionality reduction to reduce the number of sparse features. $\endgroup$
    – CyberPlayerOne
    Sep 28, 2020 at 9:48
  • $\begingroup$ What the author meant is that decision tree output fits better for sparse categories. So if there are a lot of 0s, it will try and not miss the sparsely encountered 1s. Contrast that with neural networks which won't really care about the 1s since it already got a good amount of accuracy by marking everything as 0s. $\endgroup$
    – nurettin
    Mar 1, 2021 at 11:10

1 Answer 1

Reset to default
1
$\begingroup$

There are many feature engineering options for sparse categorical data. A couple of common options:

  • Remove features - Drop columns that are sparse
  • Feature reduction - One example is singular value decomposition (SVD)
  • Reduce number of categories - Using domain knowledge, group together related low frequency categories
  • Use sparse representation - only stores nonzero elements
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.